Billing in a large project

There are various ways to “monetize” a project. But they have one thing in common - how money transfers from the user's wallet to the organization’s account. Today we will talk about how payment acceptance is organized in Badoo and what can be found on the payment gateway market. We immediately warn that in the article you will not find specific figures for the turnover of company funds, but everything else will be no less interesting.

What is billing?


For us, billing is all that is associated with receiving money from users: price configuration, payment acceptance page, directly accepting and processing payments, providing paid services, various promotions and, of course, monitoring all of the above.

Initially, as in all startups, we did not have paid services. The first serious steps towards monetization began back in 2008, despite the fact that the site was officially launched in 2006. For experiments, France was chosen, and payment was accepted only via SMS. The payment acceptance itself was organized on files. Each request was written to a separate file, which was then transferred by bash scripts from one folder to another, which meant a change in processing statuses. The database was used only to record successfully processed transactions. Such a scheme worked successfully for a little over a year, after which it became difficult to maintain, and we decided to abandon the files and rewrite everything using the database.

The development of the new version was quick enough, since there were not many countries where paid services were available. But it was designed only for receiving payments via SMS, because of this we still have several funny artifacts, for example, the MSISDN (phone number) and short code (short number to which paid SMS is sent to) fields in the processed table payments.

Now we accept payments almost all over the world. Every second, users try to pay for something on the site or in applications for all popular mobile platforms. And if you put this on the map, you get the picture "View of the Earth from space at night":

image

We have about 50 payment methods available from different partners. The most popular are bank cards, SMS & Direct billing and purchases in mobile applications.

image

Among them, there are exotic ones, for example, direct debiting from the account of an Internet provider or payment via a landline phone. And once we received a payment via regular mail!

image

Bank Payments


All payment systems allow you to accept payments from your users. It is convenient to do such direct integrations until there are a lot of them and you connect well-known systems with debugged processes. But when you need to enter local markets, then problems begin to appear. Maintaining the “zoo” of different APIs is becoming increasingly difficult, the requirements of regulators differ, the popular local payment system may even refuse to work with foreign clients at low speeds, or signing a contract and resolving legal problems can take a long time. Despite such difficulties, local payment systems can pleasantly surprise you with their conversion. For example, the Netherlands, which we considered not very promising, after connecting the iDeal payment method popular in this country, began to bring 30-40% more money.

If there is demand, there will be an offer. There are many aggregator companies, or, in another way, payment gateways on the market . payment gateway), the purpose of which is to unite all popular payment systems, including local ones, under a single API. Having made one such integration, we get the opportunity to accept payments through dozens of payment systems around the world. You don’t even have to make a payment page on your site, but use the ready-made one provided by the aggregator and tailor only the design for yourself. Especially advanced companies make it possible to upload their CSS and JS files, change pictures, translation texts and even register the resulting page on your subdomain, for example, payments.example.com. This gives a very rich opportunity for "customization", and the user has the feeling that he is not leaving your site. We do not use this opportunity at home, since we are simultaneously working with several aggregators, но для кого-то это может быть очень удобным решением.

Which way to use - direct integration or through an aggregator - depends primarily on the size of the commission. The more your customers use the payment system, the more profitable it may be to save on commissions and connect to it directly. The second important factor is API quality, usability and stability. Here, aggregators can smooth out roughness, and sometimes provide a more stable service than a direct connection.

SMS payments


Payments via SMS and direct deductions from the balance of a mobile phone stand apart. They are under very tight control in many countries, especially in Europe. Local regulators or the state itself may have special requirements for how the payment page should look or what the text of the SMS messages should be. Changes to such requirements must be monitored and changes made on site on time. So, for example, in Belgium there is a rule that a short number should be written in white font on a black background, and its cost should be indicated next to it.

image

The type of SMS billing is different - MO (Mobile Originated) or MT (Mobile Terminated). With MO-billing, everything is quite simple: the user sent an SMS to a short number, we received the money. But for MT there are several options. Payment does not occur at the time the user sends an SMS message, but after he receives a special paid SMS from us. That is, the fact of payment is the notification received from the aggregator that the paid SMS was successfully delivered. The main goal of this approach is to add an additional check before sending a paid SMS message to the user and prevent errors associated with incorrect text in SMS. For this, payment takes place in two phases. The first is that the user expresses a desire to pay for something, the second is confirmation. For example, the payment process may look like this:
  • send an SMS to a short number, respond to an incoming SMS with a specific text or without it;
  • send SMS to a short number, enter the received PIN-code on the site;
  • enter the phone number on the website, get the PIN code, enter it on the website.

Fortunately, there are also aggregator companies in the SMS payment market, whose services are worth using. They are for a modest, and sometimes not very, fee deprive you of the "pleasure" to deal with such details. Another nice bonus is that they often take on part of the responsibility for supporting end users. Users begin to write about their problems directly to the owner of the short number, i.e. to the aggregator.

Technical details


Badoo runs on a bunch of PHP + MySQL, so we use the same technology to process payments. The code runs on a separate server group isolated from a common pool. Inside, we divided it into several logical subgroups: servers for processing incoming requests, servers for background operations and statistics collection, database servers, servers for processing bank card payments. The latter are allocated in a separate group, because they must comply with the PCI DSS security standard developed with the participation of Visa, MasterCard, American Express, JCB and Discover for organizations working or storing bank card holder data.

For payment processing, we use two database servers with MySQL from Percona, working in master-master replication. The main load goes only to one of them, the second is used for “hot” replacement in the event of an accident or to replace the main one (during its service, for requests from the monitoring system or statistics collection).

The entire billing system can be divided into several large parts:
  • Nucleus. This includes basic entities, such as Order , Payment , Services and the rules for their accounting and provision, various infrastructure things.
  • Aggregator plugins. Everything that is responsible for communication between us and the payment system.
  • Page for choosing and paying for services.

Connecting a new payment method to the system consists precisely in creating its plugin. He is responsible for all the interactions between us and the payment gateway, which are of two types: when we are the initiator (pull request) and when the initiator is the aggregator (push request). For pull requests, HTTP is usually used as the protocol, either pure or as a transport for JSON / XML or SOAP. The recently popular REST API concept has not been seen often, only among new companies on the market, for example, British GoCardless, or old companies that decided to redesign their API, for example, PayPal. For push requests, SOAP is practically not used (of those who offer this format of push notifications, only Qiwi is easily remembered). Pure HTTP is mainly used or it is also used as a transport for all the same JSON and XML.

After implementing the API, the testing phase begins. There were already articles on Habr how our process of development and automation looks . But for billing, there are some features related mainly to the fact that you have to test not just our code, but also the interaction with aggregators. It is very convenient if they have for this a test environment that fully emulates the real payment acceptance. If it is not there, we make “stubs” emulating the behavior of the aggregator. This simplifies our manual testing and allows us to write self-tests that verify the entire payment process. Here's an example of what one of the stubs looks like.

image

After the test environment, you need to check how everything will work in life, make a real payment. But for SMS payments, you often have to get approval from regulators or operators, and this can take several months. In order not to spread the semi-finished code on production servers, we came up with such a thing as External Shot. This is our usual Shot, which is a directory with a task branch and is intended for testing it on production servers, but in addition to the local domain, it has an additional external address at which anyone can go in and see the changes made. For security, such “shots” are created not for every task, but only in those cases when it is really necessary. We give links to them to our partners, and they can check the changes made at any time of the day or night. Особенно это актуально для стран, расположенных в другом полушарии, с которым разница во времени может достигать 12 часов.

Support and operation


After the new integration is laid out on production servers, the stage of its operation and support begins. Technical support takes about 60-70% of our time.

image

This includes, firstly, the analysis of user complaints. All simple situations are resolved by the team of the first line of support, it also translates complaints for us from different languages ​​into English. Therefore, we get only the most complex cases that really require the attention of developers.

The second component of technical support is fixing bugs or making changes to existing integrations. Errors occur for various reasons. For example, due to inattentive reading of documentation or spaces in it. Once, we even had to use chat logs with the developer of the aggregator instead, because the documentation for their new system was not yet ready. There were cases when the aggregator changed the interaction protocol or its parameters without notice. Another time, the acquirer bank disconnected our gateway, and we had to urgently redirect traffic to another place. As it turned out later, it was an ancient server from the 80s, which, according to the bank, should not process anything at all. In general, you don’t have to be bored, especially when you consider that every minute of downtime is a lost profit.

To solve such problems, we write detailed logs of the application. Not only errors get there, but also all interaction with aggregator systems or simply important events that occur during query execution. Each request has its own unique identifier, by which you can find all the records associated with it and restore its processing. This is especially useful when you have to deal with errors from which several weeks or months have passed.

This is how the billing in Badoo is organized. Of course, there are still many interesting topics that we plan to talk about in the future, for example, monitoring, PCI DSS certification and processing of bank card payments. If you have questions or any suggestions on the topic of future articles, welcome to comment.

Anatoly Panov
Lead Developer