O_SYNC, where has thy FILE_SYNC gone?

This morning I came across an interesting difference between RHEL 5 and RHEL 6 (and of course their CentOS brethren).

In the RHEL 5 kernel, when an application writes to a file opened with O_SYNC over NFS, those writes come across the wire as FILE_SYNC writes — ie: stable.

In the RHEL 6 kernel, when an application writes to a file opened with O_SYNC over NFS, those writes come across as UNSTABLE immediately followed by a COMMIT. To get the writes to come across the wire as FILE_SYNC, you have to use the O_DIRECT flag (either by itself or in addition to the O_SYNC flag).

This is a rather significant change and has storage-system impacts. Good for both users and implementers to be aware of.

Update: According to this kernel bug the RHEL 6 behavior listed above happens in kernels 2.6.32 to 2.6.38-rc3. Presumably it is fixed (ie: reverted to the RHEL 5 behavior) after 2.6.38-rc3, although I don’t know if its been picked up in the RHEL kernel.