I’ve seen a couple of Oracle Community and AskTom posts over the last year or two about installing anti-virus software on Oracle database servers. Usually it is because someone in security told the sysadmin or the DBA that they had to install some kind of AV software because it was required. Been there; done that. I found out the hard way that doing so was a bad idea.
“But why is it a bad idea?”, the security people ask. Don’t we want to make our servers as secure as possible?
Of course we do, but what does that mean?
From a high level perspective, my database server is most secure – regardless of what operating system I have chosen or whether I have AV software or not – when it is just a database server. It should not be providing any other network services that could potentially be exploited by or be used to transmit infected files, or be potential access paths for criminals. That means:
- No sharing files (as a server) with network clients through Samba, NFS, or Windows shares.
- No web servers like Apache or IIS running directly on the database server.
- No e-mail relays processing mail attachments on the database server (e-mail clients to send messages that might originate from the database are fine).
- Server network ports are protected by a local host firewall, ensuring that only those ports minimally required for the system to function (e.g. 22, 1521) are exposed to the network.
- No application users logging in to the database server operating system to run applications.
- The database server should never be used as web client, except to download installation or patch files directly from Oracle, Red Hat, Microsoft, etc. If possible, you should even download your patches to another system first and scan them there, before transferring them to the database server.
In short, only sysadmins and DBAs should be logging on to the actual server at all. Web service connections should be reverse proxied through a stand-alone, purpose-built web and/or application server. Other network services should also be placed on dedicated resources, far, far away from the database. The only client / user access should be over encrypted Oracle Net, and even that could be reverse proxied through Oracle Connection Manager.
If you’ve done all that, you’ve almost entirely mitigated the potential for virus infection in the first place. Where would a virus even come from? What problem would installation of AV software solve? “We don’t care,” says security, “install it anyway!”
Ok. So if I have to do this, what’s the best way? Let’s check with My Oracle Support to find out! A basic search on “anti-virus” yields several results.
We start with MOS Doc ID 220970.1, which asks, “Is it a good idea to add anti-virus software to my RAC cluster?” The answer is pretty generic, not really RAC-specific, which is why we start here (with emphasis added):
“For customers who choose to run anti-virus (AV) software on their database servers, they should be aware that the nature of AV software is that disk IO bandwidth is reduced slightly as most AV software checks disk writes/reads. Also, as the AV software runs, it will use CPU cycles that would normally be consumed by other server processes (e.g your database instance). As such, databases will have faster performance when not using AV software. As some AV software is known to lock the files whilst it scans then it is a good idea to exclude the Oracle Datafiles/controlfiles/logfiles from a regular AV scan.”
Then there’s Doc ID 165898.1:
Be aware about that when Norton Antivirus scans the datafiles they will be locked during the scan. Avoid this by excluding the datafiles related directories in the scan.
and Doc ID 782354.1, which gets a little more specific and warns that the following file types – not just datafiles – should be excluded from all AV scans:
- Oracle datafiles
- Control files
- Redo-log files
- Archived redo-log files if database is in archive log mode
- Files with extension ‘.ora’
- Password file
- Files with extension ‘ .log’ under ORACLE_HOME
- Files in ADR directory
- Oracle Software directories (including Oracle_HOME and Oracle base)
That doesn’t sound so bad, right? A little performance overhead is to be expected, and I just have to exclude my database files, right? Everyone is in agreement that as long as certain files are excluded from scanning, then all will be OK.
Not so fast. Let’s dig a little deeper. We start to see more serious warning signs in MOS Doc ID 1494011.1 (emphasis added):
Oracle does not certify its software with Third Party Antivirus Software. If using Third Party Antivirus Software you should thoroughly test this yourself to ensure its safe to use. Any issues encountered and raised with support may be required to be reproduced on a server that does not have the Third Party Antivirus Software installed.
That’s a little more serious. Now it sounds like by installing AV software I could potentially be running Oracle in an unsupported configuration. I could be asked to reproduce issues on a server without AV software before Oracle will help me resolve future issues! The experts from AskTom.oracle.com also reference the same Doc ID, making the same point, when answering this “should I or shouldn’t I” question for a user: https://asktom.oracle.com/pls/apex/f?p=100:11:::NO::P11_QUESTION_ID:9538960000346328149.
For a lot of people that’s a show stopper; not everyone has the resources – virtual or physical – or the time to rebuild and reload a server from scratch just to duplicate an issue for support. The AskTom expert, Connor McDonald, goes on to say, “My opinion in a nutshell – if I’m running an antivirus product on my server, I would not let it anywhere near *any* file that is managed by the Oracle database. Because any kind of spurious file locking by the AV software could lead to hangs, errors or even instance crashes.”
A good question to ask here is why doesn’t Oracle certify their software with major anti-virus software suites? It turns out – not surprisingly – that Oracle’s software is engineered for high performance. In order to do what it does, it needs to be able to do things that a lot of other types of software wouldn’t necessarily do. Aside from the issue with file locking, this also makes it extremely sensitive to other programs that require a lot of system resources themselves, that effectively throttle system resources, or that limit access to storage devices.
To understand how this resource limitation happens, we need to understand how AV software prevents other software from accessing storage media like disk drives without being scanned. In order to be as integrated as possible, AV software typically installs a module directly into the operating system kernel, effectively becoming a part of it. This allows it to see everything going on and to scan various parts of the OS like storage drivers (which also operate at the kernel-level) as they are running, and to protect against kernel-level infections of device drivers, root kits, and the like. It is also what allows the AV software to defend itself – preventing itself from being deleted – and why you generally have to reboot a system after installing or uninstalling any such software.
Because Oracle only certifies their software on very specific kernel configurations, it cannot say for sure how their software will react on a modified, or “tainted” kernel that includes untested 3rd party modules. Oracle takes this seriously enough that MOS Doc ID 284823.1 says:
Oracle Linux environments in which customers recompile the kernel, insert third-party provided kernel modules, or recompile glibc are not eligible for Linux kernel support.
So if you’re using Oracle Linux, installing AV software may potentially invalidate your operating system support in addition to your database support.
If that still doesn’t deter you, what sort of behavior might we see on a server with AV software loaded? Here’s a few examples of issues that have been reported over the years:
- MOS Doc ID 2215889.1 describes a case where AV software was interfering in the Windows kernel enough to cause lsnrctl and tnsping to core dump.
- MOS Doc ID 1957142.1 and Doc ID 2549476.1 describe how some NFS mounts stopped working after AV software was installed on a Linux system.
- My personal favorite, MOS Doc ID 789895.1 describes how AV software blocked file access on Linux when using Direct I/O. Direct I/O bypasses local file systems and lets Oracle submit I/O requests directly to the kernel (through its own kernel module) and the storage media drivers.
In each case, the only work-around was to remove the AV software entirely.
As you might have guessed I’ve had some experience with that last one. Using two different, commercial Linux AV software suites and two different Linux kernel versions (Red Hat Enterprise 6 and Oracle Linux 7), I saw exactly the same issues. The AV software fouled up my RAC clusters consistently by blocking the Direct I/O and throttling my interconnect network to the point that it was causing rolling node evictions and server reboots (every five minutes!), datafile corruption, and in general making the entire system unstable to the point of being completely unusable. It was not possible to exclude the affected drivers and network ports from scanning with either AV suite.
While Direct I/O can be disabled in some circumstances, it is all but required when using raw partitions or Oracle ASM (which is in turn required by Oracle RAC), and is the default access method for Linux in general because it reduces memory overhead and increases performance for most systems. It is also used extensively by RMAN and DataPump, and by various Oracle engineered solutions and database appliances. Yet for some reason AV software seems to block it on principle, as it perceives the direct kernel requests from Oracle to be a malicious attempt to bypass normal file system access and AV scans.
Even if you can disable Direct I/O in your specific configuration, doing so may impact your performance drastically (users have reported gains of 30-40% using Direct I/O). Again, my personal experience has been that the “slight” reduction in bandwidth and other resources described previously doesn’t really do justice to the reality, which is more often a severely impaired system, regardless of your operating system.
If you’re thinking that this sort of thing will get sorted out sometime soon, don’t count on it. Oracle and McAfee (and other AV vendors) have been disagreeing over this for something like 15 or 20 years already – Oracle calls the Direct I/O block a bug, and McAfee calls it a design feature, protecting against unrecognized kernel modules – and there’s no end in sight.
So where does that leave us? You will have to test the final effects of any AV software in your own specific architecture to be sure how it will react, but in summary:
- Just by installing it you may void your support for both the Oracle database software and (if using Oracle Linux) your operating system kernel. At the very least you may have to rebuild your system without AV software to get help.
- Even if you are not planning to use Oracle ASM and you are able to disable Direct I/O support, using normal file system access will be slower in most cases (possibly by orders of magnitude).
- Even with features like Direct I/O disabled and Oracle-related files excluded from scanning, AV software may still impact the overall performance of your system to the point where processes crash, and data becomes corrupted.
Finally, what about those of us who work under the purview of the Defense Information Systems Agency, who’s Secure Technical Implementation Guides (STIGs) dictate both that AV software will be installed and that all software must be run in supported configurations? It is literally impossible to reconcile these two requirements, especially when the AV software is centrally managed, as it is on AFNET, and new policies and rules can be deployed at any time without warning.
So how do you deal with this conundrum in a practical way that keeps everyone happy? The first thing to do is to try and get a waiver for your database server from your responsible Information Assurance organization that allows you to operate it without any AV software installed. This may be a long shot at best, but it has been done before. You are more likely to get a waiver if you can make the following points about your database server by following the guidelines at the top of this article:
- It is a Linux server (or at least not Windows).
- Only administrators have access to log on to the server operating system directly – not developers, and not definitely not end users.
- The server does not directly provide any file sharing services such as Samba, NFS, or Windows file shares to Windows clients.
- The server does not provide any other network services, such as HTTP, that users can interact with directly.
- User interaction with the database is through means of a separate web/application server tier that does run antivirus software.
- Your system architecture requires the use of ASM storage, which is completely incompatible with AV software.
Clearly the use of AV software on a database server is not simple or straight-forward. In general – and especially if you can secure things as recommended at the start of this post – I would avoid it at all costs. I have successfully used the issue with Direct I/O as sufficient cause to justify complete exclusion of AV software itself from my database servers in some of the most demanding security environments anywhere. If you still can’t avoid it, then limit its scope as much as you can. Make sure you exempt anything Oracle-related from both active and scheduled scanning, and be prepared to use little tricks like the chattr immutable file flag on Linux to make sure that your exemptions stay that way and don’t get overwritten by some overzealous system or network administrator, or an uncaring group policy pushed by others.
Always be responsible for the safety, security, and performance of your servers, and be prepared to be the voice of reason in the face of irrational fears.