[Pkg-samba-maint] Bug#641598: samba: Directory listings fail under some situations with large number of files
Erik Meitner
erik at wanderings.us
Wed Sep 14 15:15:41 UTC 2011
Package: samba
Version: 2:3.2.5-4lenny14
Severity: important
The folder contains 146 directories and 26461 files at the top level. When listing the folder contents only 4616 items are listed.
My testing shows this happens with all Linux Samba client implementations I have access to:
SMBClient under Ubuntu/10.04, Debian/Squeeze, Debian/Lenny, Debian/Etch
GVFS Under Ubuntu/10.04
It happens from the server itself using the command line client. It does NOT happen with a Windows XP client.
Note that some information has been redacted as it contains names and member numbers of our customers.
What I found is that Linux Samba clients are reporting an error at some point due to a certain file:
Error: Looping in FIND_NEXT as name xxxxx_gggggggg, Amy.pdf has already been seen?
Using the Samba command line client I see duplicate entry during listing of files:
...
xxxxx_gggggggg, Amy.pdf 396332 Fri Jun 17 11:44:35 2011
xxxxx_gggggggg, Amy.pdf 396332 Fri Jun 17 11:44:35 2011
...
The above two lines are the 4499th and 4500th in the list.
The only other copy of the file in that folder is in a subfolder:
# find . -name '*xxxxx*' -print0 |xargs -0 ls -l
-rw-rw-r-- 1 root DomainUsers 396332 2011-06-17 11:44 ./xxxxx_gggggggg, Amy.pdf
-rw-rw-r-- 1 l.olson DomainUsers 396332 2010-07-02 18:01 ./X St. Co-op PDF Membership Disc 1/xxxxx_gggggggg, Amy.pdf
I verified that there are no binary characters in the file name.
If I remove the file the problem ceases.
If I replace it with an empty but identically named file the error happens.
The error does not happen if I alter the file name with prefixes or suffixes.
I can prevent the problem from happening by creating an additonal file named: "xxxxx_gggggggg, AmyA.pdf". Note the "A" added to the name. Creating "xxxxx_gggggggg\,\ AmyZ.pdf" did nothing.
If I create another folder with 27,000 files(named 1 though 27000) and 200 folders(named d1 though d200) and list its contents the problem does not happen. If I create an additional file in that folder named "xxxxx_gggggggg, Amy.pdf" the problem does not happen.
I created the identical directory structure as the problematic folder with the exception that all the files are empty. The command line client fails in an identical way, on the same file. I continued to test on this directory because it was not production data.
If I move some directories away the problem ceases. Removing one directory with 65 files in it caused the problem to stop. Removing only the files in that directory did nothing. Removing some other directories does nothing. Whether a directory has this effect does not seem to depend on whether it is before or after the duplicate entries.
As far as I can tell it has to do with the size of data being sent rather than the count of files or folders. I can remove the first entry returned from the 'dir' command and the error does not happen the next time. If only the 8th item is removed the error does happen.
I copied the test folder to another Lenny server with the same Samba version but a different smb.conf config. I was *NOT* able to reproduce the problem there.
Other notes:
Samba shares are on an ext3 file system.
The server where the problem happens is a production server so there are limits to the kind of testing I can do.
I am willing to send the following information but not publicly becase the filenames themselves contain names and membership numbers of our customers:
tcpdump capture of communications with server
tarball of the directory(with empty files)
debug log(-d10) from smbclient
debug log from server(please let me know what how you want this collected)
-- System Information:
Debian Release: 5.0.8
APT prefers oldstable
APT policy: (500, 'oldstable')
Architecture: amd64 (x86_64)
Kernel: Linux 2.6.26-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash
Versions of packages samba depends on:
ii adduser 3.110 add and remove users and groups
ii debconf [debcon 1.5.24 Debian configuration management sy
ii libacl1 2.2.47-2 Access control list shared library
ii libattr1 1:2.4.43-2 Extended attribute shared library
ii libc6 2.7-18lenny7 GNU C Library: Shared libraries
ii libcomerr2 1.41.3-1 common error description library
ii libcups2 1.3.8-1+lenny9 Common UNIX Printing System(tm) -
ii libgnutls26 2.4.2-6+lenny2 the GNU TLS library - runtime libr
ii libkrb53 1.6.dfsg.4~beta1-5lenny6 MIT Kerberos runtime libraries
ii libldap-2.4-2 2.4.11-1+lenny2 OpenLDAP libraries
ii libpam-modules 1.0.1-5+lenny1 Pluggable Authentication Modules f
ii libpam-runtime 1.0.1-5+lenny1 Runtime support for the PAM librar
ii libpam0g 1.0.1-5+lenny1 Pluggable Authentication Modules l
ii libpopt0 1.14-4 lib for parsing cmdline parameters
ii libtalloc1 1.2.0~git20080616-1 hierarchical pool based memory all
ii libwbclient0 2:3.2.5-4lenny14 client library for interfacing wit
ii logrotate 3.7.1-5 Log rotation utility
ii lsb-base 3.2-20 Linux Standard Base 3.2 init scrip
ii procps 1:3.2.7-11 /proc file system utilities
ii samba-common 2:3.2.5-4lenny14 Samba common files used by both th
ii update-inetd 4.31 inetd configuration file updater
ii zlib1g 1:1.2.3.3.dfsg-12 compression library - runtime
samba recommends no packages.
Versions of packages samba suggests:
pn ldb-tools <none> (no description available)
pn openbsd-inetd | inet-superser <none> (no description available)
ii smbldap-tools 0.9.4-1 Scripts to manage Unix and Samba a
-- debconf information:
samba/run_mode: daemons
samba/generate_smbpasswd: true
*** /tmp/smb.conf
[global]
workgroup = PDC
netbios name = Xstreet
netbios aliases = genie essvr
server string = 'East-side file server'
wins support = yes
wins proxy = no
dns proxy = yes
name resolve order = wins lmhosts host bcast
interfaces = lo 10.1.1.20/24
bind interfaces only = yes
log file = /var/log/samba/pdclog.%m
log level = 1
max log size = 1000
syslog only = no
syslog = 0
panic action = /usr/share/samba/panic-action %d
socket options = TCP_NODELAY SO_RCVBUF=32768 SO_SNDBUF=32768
domain master = yes
local master = yes
os level = 64
preferred master = yes
security = user
encrypt passwords = true
passdb backend = ldapsam:ldap://localhost/
ldap suffix = dc=xstreet,dc=coop
ldap machine suffix = ou=Computers
ldap user suffix = ou=People
ldap group suffix = ou=Groups
ldap admin dn = cn=admin,dc=xstreet,dc=coop
ldap passwd sync = yes
ldap delete dn = yes
add machine script = /usr/sbin/smbldap-useradd -w %u
obey pam restrictions = yes
domain logons = yes
;user profiles and home directory
; %a gives per architcture profiles for users
logon home = \\%L\%U\.%a-profile
logon drive = H:
logon path = \\%L\profiles\%U-%a
logon script = logon.bat
logon script = %U-logon.bat
load printers = no
#======================= Data Share Definitions =======================
[homes]
comment = Home Directories
browseable = no
writeable = yes
path = /faserv/homes/%U
hide files = /desktop.ini/ntuser.ini/NTUSER.*/
[netlogon]
path = /faserv/netlogon
read only = yes
write list = @DomainAdmins
browseable = no
force create mode = 775
force directory mode = 775
public = no
guest ok = yes
browseable = yes
locking = no
root preexec = /faserv/netlogon/logon.pl /faserv/netlogon/ %U
comment = Network Logon Service
[profiles]
comment = Users profiles
path = /faserv/profiles
guest ok = no
writeable = yes
browseable = no
create mask = 0600
directory mask = 0700
profile acls = yes
csc policy = disable
hide files = /desktop.ini/ntuser.ini/NTUSER.*/.*/DesktopFolderDB/TrashFor%m/resource.frk/
[allusers]
browseable = no
path = /faserv/homes
read only = No
valid users = administrator
###########################
# General shares
###########################
[sysadmin]
comment = IT Stuff
path = /shares/sysadmin
read only = No
valid users = @sysadmin Administrator
force group = +sysadmin
force create mode = 0660
force directory mode = 0770
[it]
comment = IT Cruft
path = /shares/it
read only = No
valid users = @it Administrator copier
force group = +it
force create mode = 0660
force directory mode = 0770
[genie]
path = /shares/genie
valid users = @DomainUsers
read only = No
create mask = 0664
force create mode = 0664
directory mask = 0775
force directory mode = 0775
[marketing]
path = /shares/genie_marketing/
valid users = @DomainUsers Administrator
read only = Yes
write list = @communications Administrator
force group = +communications
force create mode = 0664
force directory mode = 0775
[Art]
path = /shares/art
read only = yes
valid users = @DomainUsers Administrator @it
write list = @communications Administrator @it
force group = communications
force create mode = 0664
force directory mode = 0775
[hr]
path = /shares/hr/
valid users = @humanResources Administrator copier
read only = no
force group = humanResources
force create mode = 0660
force directory mode = 0770
[benefits]
path = /shares/benefits/
valid users = @humanResources Administrator
read only = no
force group = humanResources
force create mode = 0660
force directory mode = 0770
[ALL-shiftreports]
path = /shares/shiftreports
valid users = Administrator @managers @mgrHR @mgrGeneral @mgrES @mgrWS
read only = No
force group = root
force user = root
force create mode = 0660
force directory mode = 0770
[ES-shiftreports]
path = /shares/shiftreports/ES
valid users = Administrator @asstmgrDeliES @asstmgrGenMerchES @asstmgrGroceryES @asstmgrProduceES @managers @mgrDeliES @mgrFE-ES @mgrGeneral @mgrGenMerchES @mgrGroceryES @mgrHR @mgrKitchen @mgrOD-ES @mgrProduceES
copy = ALL-shiftreports
[WS-shiftreports]
path = /shares/shiftreports/WS
valid users = Administrator @asstmgrDeliWS @asstmgrGenMerchWS @asstmgrGroceryWS @asstmgrProduceWS @managers @mgrDeliWS @mgrFE-WS @mgrGeneral @mgrGenMerchWS @mgrGroceryWS @mgrHR @mgrKitchen @mgrOD-WS @mgrProduceWS
copy = ALL-shiftreports
[applicants]
path = /shares/applicants
valid users = Administrator @managers @mgrAllGMT @mgrAllProd @mgrAllNonProd
read only = No
force group = root
force user = root
force create mode = 0660
force directory mode = 0770
######################
# Evals shares
######################
#
# ES Store
#
[ES-FE-evals]
path = /shares/evals/ES/Front-End
valid users = @mgrES @mgrFE-ES @humanResources Administrator
read only = No
security mask = 0770
directory security mask = 0770
force group = DomainUsers
force user = root
force create mode = 0660
force directory mode = 0770
[ES-Deli-evals]
path = /shares/evals/ES/Deli
valid users = @mgrDeliES @asstmgrDeliES @humanResources Administrator @mgrES
copy = ES-FE-evals
[ES-Produce-evals]
path = /shares/evals/ES/Produce
valid users = @mgrProduceES @asstmgrProduceES @humanResources Administrator @mgrES
copy = ES-FE-evals
[ES-Grocery-evals]
path = /shares/evals/ES/Grocery
valid users = @mgrGroceryES @asstmgrGroceryES @humanResources Administrator @mgrES
copy = ES-FE-evals
[ES-GenMerch-evals]
path = /shares/evals/ES/GenMerch
valid users = @mgrGenMerchES @asstmgrGenMerchES @humanResources Administrator @mgrES
copy = ES-FE-evals
[ES-evals]
path = /shares/evals/ES
valid users = Administrator @humanResources @mgrES @mgrGeneral
copy = ES-FE-evals
#
# Ws Store
#
[WS-FE-evals]
path = /shares/evals/WS/Front-End
valid users = @mgrFE-WS @humanResources Administrator
copy = ES-FE-evals
[WS-Deli-evals]
path = /shares/evals/WS/Deli
valid users = @mgrDeliWS @humanResources Administrator @mgrWS
copy = ES-FE-evals
[WS-Meat-evals]
path = /shares/evals/WS/Meat
valid users = @mgrMeatWS @humanResources Administrator @mgrWS
copy = ES-FE-evals
[WS-Produce-evals]
path = /shares/evals/WS/Produce
valid users = @mgrProduceWS @asstmgrProduceWS @humanResources Administrator @mgrWS
copy = ES-FE-evals
[WS-Grocery-evals]
path = /shares/evals/WS/Grocery
valid users = @mgrGroceryWS @asstmgrGroceryWS @humanResources Administrator @mgrWS
copy = ES-FE-evals
[WS-GenMerch-evals]
path = /shares/evals/WS/GenMerch
valid users = @mgrGenMerchWS @asstmgrGenMerchWS @humanResources Administrator @mgrWS
copy = ES-FE-evals
[WS-evals]
path = /shares/evals/WS
valid users = Administrator @humanResources @mgrWS
copy = ES-FE-evals
#
# Admin
#
[FoG-evals]
path = /shares/evals/FoG
valid users = Administrator @humanResources @mgrFOG @mgrPurchasing @coordFOG
copy = ES-FE-evals
[Communications-evals]
path = /shares/evals/Communications
valid users = Administrator @humanResources @mgrCom @mgrGeneral
copy = ES-FE-evals
[CoopServ-evals]
path=/shares/evals/CoopServ
valid users = Administrator @humanResources @mgrCoopServ @mgrGeneral
copy = ES-FE-evals
[HR-evals]
path = /shares/evals/HR
valid users = Administrator @mgrGeneral @mgrHR
copy = ES-FE-evals
[Finance-evals]
path = /shares/evals/Finance
valid users = Administrator @humanResources @mgrGeneral @mgrFinance
copy = ES-FE-evals
[IT-evals]
path = /shares/evals/IT
valid users = Administrator @humanResources @mgrIT @mgrGeneral
copy = ES-FE-evals
[GMT-evals]
path = /shares/evals/GMT
valid users = Administrator @humanResources @mgrGeneral
copy = ES-FE-evals
[Operations-evals]
path = /shares/evals/Operations
valid users = @mgrOperations @humanResources @mgrGeneral Administrator
copy = ES-FE-evals
[Maintenance-evals]
path = /shares/evals/Operations/Maintenance
valid users = Administrator @humanResources @maintCoord @mgrOperations @mgrGeneral
copy = ES-FE-evals
#
# OSK
#
[OSK-evals]
path = /shares/evals/OSK
valid users = Administrator @humanResources @mgrGeneral @mgrKitchen
copy = ES-FE-evals
[Catering-evals]
path = /shares/evals/OSK/Catering
valid users = Administrator @humanResources @mgrCatering @mgrKitchen @mgrGeneral
copy = ES-FE-evals
[OSK-kitchen-evals]
path = /shares/evals/OSK/Kitchen
valid users = Administrator @humanResources @SousChef @mgrKitchen @mgrGeneral
copy = ES-FE-evals
[OSK-bakery-evals]
path = /shares/evals/OSK/Bakery
valid users = Administrator @humanResources @coordBakery @mgrKitchen @mgrGeneral
copy = ES-FE-evals
[OSK-FOG-evals]
path = /shares/evals/OSK/FOG/
valid users = Administrator @humanResources @coordFOGOSK @mgrKitchen @mgrGeneral
copy = ES-FE-evals
#
#Top level for HR and GM
#
[evals]
path = /shares/evals
valid users = Administrator @mgrHR @mgrGeneral
browseable = no
copy = ES-FE-evals
######################
# Utility shares
######################
[scaleupdates]
path = /shares/scaleupdates
read only = No
valid users = @DomainUsers @automator
force create mode = 0660
force directory mode = 0770
[iclupdates]
path = /shares/iclupdates
read only = No
valid users = @DomainUsers @automator
force group = DomainUsers
force create mode = 0660
force directory mode = 0770
[drops]
path = /shares/drops
valid users = @DomainUsers @automator
read only = No
force group = DomainUsers
force user = root
force create mode = 0660
force directory mode = 0770
browseable = no
[onetime]
path = /faserv/jobs/onetime
valid users = @automator @it
read only = No
force group = DomainUsers
force user = root
force create mode = 0660
force directory mode = 0770
browseable = no
##################
# Temporary shares
##################
[staffphotos]
path = "/shares/hr/Staff Photos"
valid users = @communications @it
read only = yes
force group = root
force user = root
force create mode = 0660
force directory mode = 0770
browseable = no
More information about the Pkg-samba-maint
mailing list