Problems with support of file locks for ARC (maybe for other targets too)

[Yuriy Kolerov]

Hi Veneet,

> -----Original Message-----
> From: Vineet Gupta
> Sent: Wednesday, August 10, 2016 4:01 AM
> To: Yuriy Kolerov <ykolerov at synopsys.com>; uclibc at uclibc.org
> Cc: Waldemar Brodkorb <wbx at openadk.org>; Alexey Brodkin
> <abrodkin at synopsys.com>; Anton Kolesov <akolesov at synopsys.com>
> Subject: Re: Problems with support of file locks for ARC (maybe for other
> targets too)
> 
> Hi Yuriy,
> 
> On 08/09/2016 10:06 AM, Yuriy Kolerov wrote:
> > 1. uClibc does not support OFD locks (https://lwn.net/Articles/586904/).
> fcntl.h headers for all targets are not synchronized with Linux source tree and
> does not contain these defines:
> >
> > ------------------------ 8< ------------------------ #define
> > F_OFD_GETLK 36 #define F_OFD_SETLK 37 #define F_OFD_SETLKW 38
> > ------------------------ 8< ------------------------
> >
> > Also appropriate changes must be done in
> "libc/sysdeps/linux/common/__syscall_fcntl.c". I am not sure how to do it in
> the proper manner because it involves a lot of duplication for each target...
> 
> This is common generic ABI and applicable only to select few / newer arches.
> 
> >
> > 2. uClibc uses fcntl64 instead of fcntl by default:
> > http://cgit.uclibc-ng.org/cgi/cgit/uclibc-ng.git/tree/libc/sysdeps/lin
> > ux/common/__syscall_fcntl.c
> 
> FWIW - linux/common/* is the common generic syscall ABI that was added to
> uClibc when ARC and metag ports were added to linux kernel. The idea is to
> use asm-generic kernel headers and for syscalls uses min number of syscalls
> as possible (so xxx64 only would be exposed by uapi headers. For this case
> specifically, only __NR_fcntl64 will be defied in the kernel.
> 
> > Also if you want to use file locks you need to pass flock or flock64 structure
> to fcntl which are defined in fcntl.h for each target. E.g. for ARC:
> > http://cgit.uclibc-ng.org/cgi/cgit/uclibc-ng.git/tree/libc/sysdeps/lin
> > ux/arc/bits/fcntl.h
> >
> > So if fcntl64 is used by default then flock must be identical to flock64 in this
> case. That is why flock and flock64 are defined this way:
> 
> Note that given the common generic syscall ABI - userspace doesn't need to
> use struct flock64 with flock64 syscall. However the userspace struct flock
> needs to match with kernel's struct flock64. See what we do for stat syscall
> family
> (libc/sysdeps/linux/common-generic/bits/stat.h)
> 
> > ------------------------ 8< ------------------------ struct flock
> >   {
> >     short int l_type;	/* Type of lock: F_RDLCK, F_WRLCK, or F_UNLCK.
> 	*/
> >     short int l_whence;	/* Where `l_start' is relative to (like `lseek').
> */
> > #ifndef __USE_FILE_OFFSET64
> >     __off_t l_start;	/* Offset where the lock begins.  */
> >     __off_t l_len;	/* Size of the locked area; zero means until EOF.  */
> > #else
> >     __off64_t l_start;	/* Offset where the lock begins.  */
> >     __off64_t l_len;	/* Size of the locked area; zero means until EOF.  */
> > #endif
> >     __pid_t l_pid;	/* Process holding the lock.  */
> >   };
> >
> > #ifdef __USE_LARGEFILE64
> > struct flock64
> >   {
> >     short int l_type;	/* Type of lock: F_RDLCK, F_WRLCK, or F_UNLCK.
> 	*/
> >     short int l_whence;	/* Where `l_start' is relative to (like `lseek').
> */
> >     __off64_t l_start;	/* Offset where the lock begins.  */
> >     __off64_t l_len;	/* Size of the locked area; zero means until EOF.  */
> >     __pid_t l_pid;	/* Process holding the lock.  */
> >   };
> > #endif
> > ------------------------ 8< ------------------------
> >
> > Macros __USE_FILE_OFFSET64 and __USE_LARGEFILE64 are defined in
> features.h when the compiler have these defines:
> >
> > ------------------------ 8< ------------------------ #define
> > _FILE_OFFSET_BITS 64 #define _LARGEFILE64_SOURCE
> > ------------------------ 8< ------------------------
> >
> > So if you want to use locks you should always define those macros.
> 
> Perhaps you know this already: these macros are part of Large File support
> (LFS).
> Traditionally __LARGEFILE64_SOURCE means the 64 based API in explicitly
> used in code, while _FILE_OFFSET_BITS=64 means compiler magically
> converts regular 32 bit API into 64 bit - under the hood with preprocessor
> magic.
> 
> >  Moreover the whole uClibc and every application must be compiled with
> those macros. Here is my question. Is it acceptable just to compile toolchain
> itself using CFLAGS_FOR_TARGET with those macros and don’t define them
> manually in each application?
> 
> If you enable LFS support in build system (uClibc or buildroot) these macros
> will be automatically defined - no point or need to pass them as CFLAGS etc -
> infact that would be wrong IMO.

If you enable __UCLIBC_HAS_LFS__ while configuring uClibc then _LARGEFILE64_SOURCE and _FILE_OFFSET_BITS=64 will not be defined automatically. E.g. Buildroot has this generic line in Makefile for packages:

TARGET_CPPFLAGS += -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64

These defines are not placed to config.h or somewhere else. They must be defined each time you compile anything. uClibc may be built by Buildroot with these flags by default (actually it is not true at least for ARC - Buildroot must be fixed but it is not a subject of this discussion). However you must define these macros manually when building every application later.

I don’t understand this thing - is it possible to compile uClibc with full support of LFS (see macros above) and don’t use this feature in applications and everything will be ok? And is it possible to compile uClibc without _LARGEFILE64_SOURCE and _FILE_OFFSET_BITS=64 and compile applications with them and everything will be ok? Right now when we build toolchain for ARC using Buildroot uClibc is compiled without them and e.g. BusyBox is compiled with -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 ...

> So you really don't need LFS to be able to use locks - but need to make sure
> that for common generic ABI - user struct flock pairs correctly with kernel
> struct
> flock64 with or w/o LFS.
> 
> -Vineet