We have an Azure SQL Database running with Microsoft's default audit logging policy. The default policy audits "all the queries and stored procedures executed against the database, as well as successful and failed logins" (source; emphasis added). The audit logs are sent to Log Analytics for analysis and retention.
At the time we deployed this database, the default audit logging policy seemed reasonable: we certainly want to have a record of all authentication attempts, and having a record of all access and modification of data seemed desirable as well (e.g., to determine what data an attacker would've accessed in the event of a data breach).
However, as our application has scaled, we find ourselves producing absurdly large amounts of log entries when updating the database. This is not only very costly in Log Analytics but also triggers throttling by Azure SQL (LOG_RATE_GOVERNOR). Further, the audit logs contain "actual data" of database records -- e.g., executing a parameterized stored procedure that inserts a record into a table results in the log showing EXEC [procedure] (param value, param value, ...).
We're uncertain as to what the best practices are concerning logging of data access and modification. Specifically:
- Is it usually desirable to log all queries and procedure executions, or is it acceptable to restrict logging to database authentication attempts?
- If logging all queries and procedure executions, is it acceptable to end up with personal identifiers such as names and emails in the audit logs? If not, is there a standard way to exclude certain fields from being logged?
- If logging all queries and procedure executions, is it acceptable to exempt certain principals from logging? For example, would it be reasonable to exclude INSERT/EXEC queries ran by the principal for the application that handles mass insertion of records?
Note: We're not subject to any compliance frameworks in particular. Thank you in advance for your guidance!
As a bonus question: our application uses Python's SQLAlchemy and handles inserts by first creating a stored procedure and then calling it over and over to insert each record one by one. We assume this results in improved performance, but wonder if anyone has a reason why this is bad.