So it wasn’t a malicious attack that overwhelmed Ellie Mae’s servers after all. What exactly was it then?
The mortgage technology developer has backtracked from its previous explanation that a distributed denial-of-service attack overwhelmed its servers from March 31 to April 1. Ellie now says the outage “was triggered by a confluence of factors involving network, hardware, software and demand for service.”
That raises the question of how a provider of a loan origination system and ancillary technology could have been inundated with service requests at a time when originations are in the doldrums.
Industry volume was off about 50% from a year earlier in the fourth quarter and at 17-year lows in the first quarter of 2014. To be sure, Ellie Mae has tried to partially offset the industry’s headwinds by signing new customers and converting old ones from legacy technology to the software-as-a-service-based version of its LOS, with mixed results.
There are at least two plausible alternative explanations. One theory is that the way Ellie’s LOS stores data, using what’s known as a flat-file database, is increasingly becoming difficult to scale as more volume flows through the system. Another possibility is that the recent addition of new correspondent investors and expanded features to its Total Quality Loan program could have been the source of the surge in demand.
The mystery of what brought Ellie Mae’s systems to a screeching halt is far more than an academic question. The outage came at month- and quarter-end, a critical time for lenders, and delayed loan closings, which forced lenders to pay for rate lock extensions and hedging losses. Home closings funded by purchase mortgages were also delayed.
THE DDOS THEORY
The characteristics of the outage initially appeared to be consistent with a DDoS attack. But an investigation by cyber-security and digital forensics firm Stroz Friedberg concluded there was no malicious attack on Ellie Mae’s systems and confirmed that client and personal borrower data was not compromised, the Pleasanton, Calif. vendor said April 14.
This much is known: The crash occurred after Ellie Mae’s servers were overwhelmed with requests to a URL that is used to download an XML file containing a list of third-party technology vendors that integrate with the Encompass LOS via the Ellie Mae Network. The requests had the look and feel of legitimate communications, leading the company to initially suspect it was attacked by people familiar with the mortgage industry, Ellie Mae President and COO Jonathan Corr told National Mortgage News on April 3.
“It was a valid request with a normal signature; a very innocuous request, but something that created a tremendous amount of traffic,” Corr said then.
The outage affected users of both the SaaS and self-hosted versions of the Encompass LOS. Even self-hosted users must connect with Ellie Mae’s servers for authentication and to access the Ellie Mae Network, a communication medium and delivery channel for third-party underwriting services like document preparation, income and employment verification, and credit checks.
Some mortgage technology vendors are troubled by the potential long-term consequences of the incident and wonder if it will shake lenders’ faith in SaaS-based technology.
“When there’s a problem like this, it makes the customers nervous and that concerns me,” says Sharon Matthews, CEO of the electronic document management system provider eLynx. “They need to be able to rely on us…They need to know that we planned this stuff for throughput and quality with huge commitment, and when that doesn’t happen, I think everybody hurts.”
Others have been skeptical about Ellie Mae’s initial claim that the outage was consistent with a DDoS attack.
“There’s a much higher probability that they simply had internal issues related to the application and the architecture than a DDOS attack,” DocuTech Chief Strategy Officer Scott Stucky said on April 3.
THE FLAT FILE THEORY
Rather, the outage could have been a result of the architecture of Ellie Mae’s LOS, which uses a flat-file database instead of a relational one, Stucky and others have speculated.
Flat files are linear, meaning data is stored in one long string of text. Relational databases organize data into tables that allow for faster processing and multitasking. As more demand is put on a system, flat-file databases require more processing capacity and must be scaled using hardware and virtualized servers, while relational databases can be scaled using software.
“I think it’s fair to ask, did that in fact contribute to the complications of recovery?” Stucky says. “Because the amount of processing overhead necessary to crunch flat files is exponentially more than it is to crunch files in a relational database.”
Corr dismissed this notion as “absolutely, positively false.”
“The issue had nothing to do with access to, or recovery of, flat files as we were coming back up. This was a load event that happened at the Web services layer,” he said on April 3. “This was strictly related to an overwhelmed capacity, and even the calls that did take place were hitting our database.”
THE TQL THEORY
Another possible explanation for the surge in demand may be tied to Ellie’s Total Quality Loan program – particularly given Corr’s explanation that the outage was caused by a surge in requests for a file in the Ellie Mae Network.
TQL is a technology and services package that provides tamper-proof versions of borrower verification data on loans sold by correspondent lenders. Since launching a pilot with Wells Fargo in October 2011, Ellie Mae has signed four more correspondent investors to use the service: CitiMortgage in August 2012, Ocwen’s Homeward Capital in August 2013, and Stonegate Mortgage and PHH Mortgage, announced during Ellie’s 2013 year-end earnings call in February.
On March 20, Ellie Mae released a series of updates and enhancements to TQL, including new tools designed to help lenders meet ability-to-repay and Qualified Mortgage regulations. The new features include a central repository for borrower verifications, additional quality control checks and dashboards and reports to provide increased organizational and vendor oversight.