Our AWS production server is EC2 instance with 100GB disk space and 16GB RAM. The application is based on ruby on rails. From past 1 year application is running fine. Suddenly we started getting ENOSPC:no space left on device @ io_write error with production application stopped working. I kept on looking for bigger size files and I was surprised to see 70GB production.log file size.

Everything is Broke – Check All Logs

Once this error came, you can not create any new file on server, all the services were down.

Apache
psql
passenger

We have increased disk space from 100GB to 200GB but that did not help and ‘lsblk’ was still showing as 100GB.

Rails log

The last log from the rails application is – load command “rails/commands/server/server_command”. error: no space left on device – bs_fetch:atomic_write_cache_file:write.

Apache log

Verify in Log file – /var/log/apache2

Unexpected error in mod_passenger: An error occurred while receiving HTTP upload data: The timeout specified has expired (70007)
Backtrace:
in ‘void Passenger::Apache2Module::Hooks::sendRequestBody(const Passenger::FileDescriptor&, request_rec, bool)’ (Hooks.cpp:1226) in ‘int Passenger::Apache2Module::Hooks::handleRequest(request_rec)’ (Hooks.cpp:622)

[ /Tg Ser/Server.h:1079 ]: [Client 5-867] Disconnecting client with error: error reading request body: Unexpected end-of-stream (errno=-1004)

PostgreSQL error log-

[3512] FATAL: could not write lock file “postmaster.pid”: No space left on device
pg_ctl: could not start server
Examine the log output.
UTC [4893] LOG: listening on IPv4 address “127.0.0.1”, port 5432

Tried Online Solutions

passenger-config validate-install
sudo service apache2 restart
passenger status

These commands and not even ‘cd’ , ‘tab’ are working so, production server is completely broke.

Some more suggestions –

sudo sysctl fs.inotify.max_user_watches=524288

df -ih was showing inodes as 100%

Solution

Below are few solutions and suggestions.

logrotate

In config/environments/production.rb config logging to syslog, this way default logrotate tools will be used and logs will be automatically truncated.

# Use a different logger for distributed setups
config.logger = SyslogLogger.new

Check out more suggestions on best way to use logrotate on rails application from StackOverflow.

Change Config Log Level

Change the log level to warn or critical in config/environments/production.rb file, If it is set to ‘debug’ level it is going to dump lot of debug information in log file.

# when problems arise.
#config.log_level = :warn
config.log_level = :debug

Clear Log

You can clear the logs on the fly using following command,

cat /dev/null > log/production.log

Otherwise rm log/production.log would require restarting the app and sometimes creating production.log file manually.

Limit Size of Production Log

You can limit the size of production.log file in your rails application. In config/environments/production.rb add

config.logger = ActiveSupport::Logger.new(
config.paths['log'].first, 1, 100 * 1024 * 1024)

This will ensure log files will never grow bigger than 100Mb and with 1 historic log file will be kept.

With clearing production.log and re-starting all services manually, production server is up and running. Happy debugging and hope you have enjoyed reading ENOSPC:no space left on device @ io_write error.