Author: José Postiga
-
-
-
Recently I needed to tap on Laravel’s Passport programming to control how the JSON Web Tokens (JWT) were being issued. Specifically, I needed to add more claims to it (to hold more user information) and to control how the scopes were being generated. The idea was to add information like the authenticated user’s email, VAT number, account type, and, also, to forcibly add the scopes that were associated with the user’s role.
However, changing Laravel Passport’s behavior to make it happen isn’t quite obvious, and in a world were service-oriented architectures are becoming ever more common, JWT being the de facto way of carrying user’s information through multiple services, and since I couldn’t find quite a good resource to understand how to do it, I thought I could share with you how I approached and solved my own problem.
The need for control
I’m developing ITsoup, an IT support/helpdesk software system, designed to simplify and streamline all its related processes, based in a service-oriented architecture. This means that I’ll have many services that handle specific domains of work. In order to authenticate the requests, properly, I needed some way of holding users’ information in a way that could be passed around from service to service. The de facto way of doing this is with JWTs, so I started investigating how I could make those changes to the JWT generation logic.
The problem with current Laravel Passport implementation is that it only includes one identification claim for the user: it’s internal ID. That’s not of much use for me on this project. Using only the ID would require any other services to request any additional user information to the Organization Domain service’s API if they need, for example, the e-mail.
Doing that way would create an unwanted, unnecessary, dependency between this service and all others that work with the user’s data and would increase the load on this Organization Domain service as new services and/or traffic increased.
Understanding Laravel Passport’s Service Provider
Before anything, I like to always understand what’s going on under the hood. By getting a broader context of the system I need to change, I can make informed decisions on how to approach a solution.
As far as I understood, it all goes down on
Laravel\Passport\PassportServiceProvider. Here we can find theregisterAuthorizationServer()method, which is responsible for defining how Laravel instantiates theAuthorizationServerclass and registers the supported oAuth2 authorization grants. But before that happens, there’s a call to amakeAuthorizationServer()method. This method is the one that’s responsible to define how theAuthorizationServergets instantiated, and injects the proper dependencies from the Laravel’s Container!The key class to this whole thing is the
Bridge\AccessTokenRepository, which is one of the dependencies injected. Since it’s being injected via the Container, we can take advantage of it and inject our own instance of that class, instead. This class has a very special method,getNewToken(), which is called when an enabled authorization grant needs to generate a new token.Hooking to the logic flow, here, and direct it through our own implementation of the
AccessTokenRepositoryclass enables us to control how the JWT is created, the information it holds, and many other aspects of it.Building bridges
So, now that I’ve pinpointed exactly the class that I need to override, I defined that a call to the
Bridge\AccessTokenRepositoryclass, within Laravel’s Container, would return an instance of my ownAccessTokenRepositoryclass, which would extend the previous one but with the single detail of overriding thegetNewToken()method. As of this moment, I successfully routed the logic to my own class.Now, this method needs to return an implementation of the
AccessTokenEntityInterface. This interface defines how to compute an access token (a JWT, for example). Laravel’s Passport implementation is exactly the one that The PHP League’s implements. In fact, not wanting to override that default behavior is one of the main reasons that Passport doesn’t supply any simple solution around this problem. In order for being able to add more claims, and control how JWT is generated – and even the scopes associated – we need to override this exact implementation.At this point, I just needed to return an implementation of the
AccessTokenEntityInterfacethat suited my needs. There’re two methods, specifically, that I needed to override:convertToJWT()andgetScopes(). The first is the one that actually generates the JWT, and the second one is the one that compiles the scopes to associate with that JWT.So, basically, my extended
convertToJWT()method looked like this:private function convertToJWT(CryptKey $privateKey): Token { return (new Builder()) ->permittedFor($this->getClient()->getIdentifier()) ->identifiedBy($this->getIdentifier()) ->issuedAt(\time()) ->canOnlyBeUsedAfter(\time()) ->expiresAt($this->getExpiryDateTime()->getTimestamp()) ->relatedTo((string) $this->getUserIdentifier()) ->withClaim('scopes', $this->getScopes()) ->withClaim('customer_id', $this->user->customer_id) ->withClaim('vat_number', $this->user->vat_number) ->withClaim('name', $this->user->name) ->withClaim('email', $this->user->email) ->withClaim('account_type', $this->user->account_type) ->getToken(new Sha256(), new Key($privateKey->getKeyPath(), $privateKey->getPassPhrase())); }And my
getScopes()method looked like this:public function getScopes(): array { return $this->user ->roles() ->pluck('scopes') ->flatten() ->unique() ->map(static function ($scope) { return new Scope($scope); }) ->toArray(); }Now, the JWT is generated with what I defined to be the relevant user information, and can be passed around and, hopefully, reduce the need to query the Organization Domain service if the JWT itself already holds the data needed.
-
We’ve recently launched Place Checkup, a web application that allows users to check if a given place follows the WHO recommended safety and prevention measures against COVID-19. Business owners, which are not yet Infraspeak‘s customers, can register through this website, claim their places and register how and what measures they’re actually following. In the end, they get an A, B, or C badge qualifying their places.
Since Infraspeak’s core software business is directly related to infrastructure maintenance management, and since we develop with an API-first approach, we already had the core functionalities available to easily use on Place Checkup. Besides that, and to avoid having multiple accounts spread over different projects, we defined that we’d leverage our existent API authentication, and centralize all the accounts there.
Our API uses Passport, an oAuth2 server implementation for Laravel projects, so by sending a user’s credentials to the API service, if they are valid, it returns an encoded JWT with the user information we’d need. To keep things simple, and to avoid having to implement a Machine-to-Machine authentication, we wanted to simply store the generated JWT in session, on Place Checkup, for further usage on subsequent calls to the API.
Laravel has a very good, and extendable, authentication system, referenced throughout the documentation as Guards. Their job is to know how an authentication request gets processed and how to communicate with a Provider, whose responsibility is to know how to get the requested data from whatever persistent layer it handles.
If correctly implemented, a custom Guard would allow us to tap into middlewares, facades, and helper methods commonly used within Laravel to handle authentication, like the
Authfacade, or getting the currently authenticated user via$request->user(), or even use theauthmiddleware to secure access to private routes. That looked exactly what we needed for this project!But adding a custom Guard and Provider proved to be a little more difficult than what the documentation shows to be. Reading through the sections that describe how Guards and Providers work, I could only see how to register the custom classes to the Container, which interfaces to implement, and which type of classes it needs to return. I couldn’t quite get a clear picture of how the flow was, so I realized I’d need to dive deep into the authentication system and try to map it from within.
Understanding Guards
Laravel’s documentation describes Guards as the following:
Guards define how users are authenticated for each request.
But, what does that really mean? That was the question I had on my head while deep on the
SessionGuardclass – a preexistent Guard shipped by default with the framework – trying to understand what’s going on under the hood. I figured that a Guard has the sole responsibility of receiving requests to authenticate a user and to call the configured Provider to fetch that user on the persistent layer.This means that, for example, when we call
Auth::attempt()to attempt a login action, Laravel will defer that call to the same method on the configured Guard. This Guard will, in turn, prepare all it needs to send that request through the related Provider, which may (or may not) return an instance of theAuthenticatableinterface, the object that represents the authenticated user.I’ve also found that we actually have two interfaces for Guards:
- The
Guardinterface, referenced in the docs, aimed to a stateless authentication, like APIs; - The
StatefulGuardinterface, that extends the previousGuardinterface but also defines how to persist the authentication information in the configured session storage.
The fundamental difference between them: state management. The knowledge of the existence of this
StatefulGuardinterface pointed me to a good direction of how I could develop a Guard that would persist the JWT in the session that integrates perfectly with the expectations of Laravel itself. I was finally in a good place on understanding the key concepts of the whole authentication system.Understanding Providers
Laravel’s documentation describes Providers as the following:
Providers define how users are retrieved from your persistent storage.
Looking at the two default providers that Laravel supports,
EloquentUserProviderandDatabaseUserProvider, I noticed that first returns an instance ofModel(an Eloquent model class) and the other handles direct calls to a database table and returns an instance ofGenericUser(a slight variation of a Value Object). Both follow the sameUserProviderinterface.I concluded that Providers are an implementation of the Repository pattern, and exist only to abstract away how you validate the user’s credentials and fetch their data on whatever persistent layer it’s stored in.
Actually, that’s probably why the user’s credentials are passed on inside an array structure: because Laravel doesn’t want to be opinionated towards how you authenticate your users. You can use a combination of e-mail and password, or a one-time hash (like magic links sent to an e-mail), or whatever you want to use. It’s pretty clever, actually!
But it’s basically this… It’s inside a Provider that you actually define what data you need to authenticate a user (the credentials) and how you communicate with the persistent layer to get the relevant data. No matter how that’s done, a Provider must return an object that implements the
Authenticatableinterface. This is to ensure that Laravel uses the correct methods when working with that object, directly, like when callingAuth::id().Making all parts work together nicely
I’m pretty sure that at this point you already have a clearer understanding of the role Guards and Providers take within the Laravel’s authentication workflow: Guards receive the request to authenticate a user and pass the credentials to the defined Provider. The Provider, in turn, queries the underlying persistent storage about the relevant user data associated with those credentials and returns it all the way back to the Guard, which makes that information available for usage.
But even if you create all the required classes, you still need to register them within Laravel’s Container, so that it knows how to call those classes:
- To register the Provider, we use the
Auth::provider()method and define how the framework should instantiate the custom provider class; - To register the Guard, we use the
Auth::extend()method and define how the framework should instantiate the custom Guard.
Let me show you exactly what we’ve done to register the custom Guard and Provider used on the Place Checkup project:
public function boot(): void { Auth::provider('infraspeak-users', static function (Application $app, array $config) { return new InfraspeakUserProvider($app->make(InfraspeakApiService::class), $config); }); Auth::extend('infraspeak-jwt', static function (Application $app, $name, array $config) { return new JwtSessionGuard( Auth::createUserProvider($config['provider']), $app->make('session.store') ); }); }The
InfraspeakUserProviderclass has all the code to communicate with our API service and validate a user’s credentials and returns an instance of theAuthenticatableinterface. TheJwtSessionGuardclass has the code that takes that object, from the Provider, and persists it to the session, avoiding new calls being dispatched to the API service while that session is active and valid, protecting our API service from being flooded with authentication requests every time a protected endpoint is being processed.The very final step in all this work was to update the
config/auth.php, and switch the default Guard and Provider used for the web routes:'guards' => [ 'web' => [ 'driver' => 'infraspeak-jwt', 'provider' => 'infraspeak-users', ], ],Now we could use Laravel’s native classes to authenticate a user and use our API service to serve as the data repository to get that information.
- The
-
Schedule
- 18h30 – 18h40: Welcome introduction
- 18h40 – 19h10: Testing back and testing forward – António Fernandes, Senior Backend Developer @ Infraspeak
- 19h10 – 19h40: Integrating Laravel 7 with Telegram and WhatsApp – Vinícius Melo, Backend Developer @ VC Cloud
- 19h40 – 20h00: Q&A and Networking
-
Wow! That’s the perfect word to describe my 2019. So far, my best year ever!
At Work
My 2019 started with me leaving TBFiles and joining Infraspeak. The change came as a surprise, even to me. I was not looking for a job change, but a casual visit to the Infraspeak’s HQ, to discuss the development of a Laravel related event, which led me to meet a lot of my (soon-to-be) colleagues and the company’s vision, goals, and culture, impressed me so much that I got stuck with the feeling that I belonged there.
My work, at Infraspeak, ranged from developing new, and exciting, features, to help to improve the overall code quality, improving the development workflow and stack used and implementing a Continuous Integration pipeline. I’ve been learning and growing so much! It’s definitely the best place to work, right now. Consider applying to one of our job openings.
There’s a lot more I plan to do here, in 2020, which I’ll blog about on Infraspeak’s Tech Blog.
Open Source
I did less open-source work than what I intended. In fact, I didn’t even complete 2019’s Hacktoberfest, with only two PRs made…
However, I contributed to the akaunting/akaunting and spatie/laravel-permission projects, created a side project that I’m still actively developing (SpeakHub) and I’ve joined the VOST Portugal organization, where I’ve been helping in the API layer development and maintenance.
For 2020, I’ll be releasing a beta version of my side project and dedicate more time to Open Source contributions. I’ll definitely not miss this year’s Hacktoberfest challenge!
Community Events
I tried something new in this area: I organized a Meetup around the Portuguese Laravel community. I tried to organize another one, but several things happened that made it impossible to create a second event. In 2020, however, there will be a new edition of this event.
I’ve also attended some events, too. I was present in the first Laracon Madrid, representing the Infraspeak Engineering Team. I met a lot of new people there, got to personally thank some of my favorite creators, like Christoph Rumpel, Freek Van der Herten and Adam Wathan. I got to be with other Laravel Portugal members and even fellow podcasters Nuno Maduro, Caneco and Bruno Falcão. I’ve also attended TechInPorto, a tech-related event in the beautiful Porto city.
For 2020, I’ll aim to speak on one conference, I’ll try to attend not only TechInPorto 2020 but also Laracon Madrid 2020 and I’ll definitely organise the second edition of Laravel Portugal meetup.
Publications and Podcasts
I’ve published three articles and twenty-one journal entries. It was amazing getting to write and share this much information with the developers’ community.
I’ve also been invited to participate in the amazing Ubuntu Portugal Podcast, where I discussed my transition from the Apple ecosystem to Ubuntu. I had a very good and friendly talk with the hosts Diogo Constantino e David Negreira.
Unfortunately, the Laravel Portugal Podcast only had one episode recorded in the whole year. It’s so sad that we (me and the other hosts) didn’t get enough time to do it more often. I always believed that a Portuguese podcast about the Laravel ecosystem makes sense to exist.
For 2020, I’ll definitely try to resurrect the Laravel Portugal Podcast, making it a once or twice per month event. Also, I’ll also make a weekly journal entry to round-up about work done and other relevant events. Maybe I’ll make a 2-in-1 and we might have a weekly Portuguese podcast about that. Who knows…
Final Words
This last year was a year full of experiments and trials and errors. As I close 2019 as the best year ever, so far, I look for 2020 with a lot of expectations to surpass the previous one. I’ll definitely begin to see the results of my work in Infraspeak, and will channel all my experience gathered until now towards open source and community development.
Wishing you all the best for this new year!
-
Yeah, it sucks.
Last week I tried to contribute to Mohammed Said’s Wink Laravel package (https://github.com/writingink/wink) in order to make it use the Laravel’s default auth system, since Wik uses its own guards, register and login workflow.
After a couple days of work, and a nice discussion on the PR, Mohammed though that it was best not to merge the PR because he wanted it to be a more flexible solution as my PR would force Wink to use Laravel’s default database connection and would update the users table with Wink specific columns.
Although the feeling of rejection sucks, I thank Mohammed, and others that commented on my PR, as they allowed me to grow and learn that contributing with open source is not all about quantity of PRs merged, but the quality of the products afterwards.
But this isn’t over, yet. We’re still discussing a better approach to this situation. If you’re interesting what we’re planning, check the issue on GitHub: https://github.com/writingink/wink/issues/13.
-
This article was originally submitted on the Infraspeak tech blog.
The hosts file is used by the system’s DNS resolver to map a fully qualified domain name (FQDN) to its related IP, without the need to query any of the Internet’s DNS servers. However, since updating this file, manually, by the common computer user would be impractical, its common use is to map the local IP 127.0.0.1 to localhost domain so that it resolves to the host machine.
So, since this file overrides the default behavior of querying the Internet’s DNS servers, and directly maps any string of characters to an IP, it’s frequently used by programmers to associate a development domain with the localhost IP address. This enables the use of a non-existent domain like, for example,
my-secret-project.testas an FQDN on a browser, as you’d normally access a registered.com(or any other TLD) domain.At Infraspeak, we have several projects that use personalized domain names in our development environment. This requires that each of those domain names have to be mapped in the hosts file of every developer’s computer. Every time we use a new, or change an existent, project domain name, we have to warn everybody about it, and each developer has to update the hosts file.
Since we’re investing a lot in hiring, it got me thinking about the tedious work of every new colleague editing the same file and paste the same mappings over and over again, and I decided to investigate a way of automating the process. I’ve come across several infrastructure-related projects, in my past, that does this. Valet, for example, which is a Laravel development environment, does this exact thing: it routes all requests for domains that use the
.testTLD to a preconfigured folder with the same name of the domain requested. So, a request tomy-secret-project.testwould route to a project inside a my-secret-project folder.I knew that Valet leveraged Dnsmasq to do that, so I went to read more about how that software works and how could I add it to my stack and route the development domains, without editing the hosts file. While following the documentation to install it in my machine, I got an error related to port 53 being already in use. While trying to find which service was using that port, I found that a package named
dnsmasq-basewas already installed on my Ubuntu machine.I don’t really know what Ubuntu is using that package for, or if it was really used at all, but I guess that it could be that the NetworkManager was leveraging the capabilities of DNS caching that Dnsmasq has. Anyway, the fact that it was already installed saved me some steps on configuring the environment. Now, I just needed to find a way to configure the Ubuntu NetworkManager to use Dnsmasq instead of system-resolve.
A little more time investigating showed me that NetworkManager already supports Dnsmasq out-of-the-box and that to enable its use all that’s required is a single key in the
[main]section of its configuration file. That’s simple enough, but I’ve been working with Linux for a lot of time now, to know that almost every system’s service has aconf.dfolder that’s used to override the default configuration with the use of partial overriding of variables. Without any surprise, I found that the NetworkManager was no exception to this.That meant I simply needed to add a file to the
conf.dfolder with only the configuration needed to activate the Dnsmasq support. I called the filednsmasq.confand put it inside the folder/etc/NetworkManager/conf.d, with the following content:[main] dns=dnsmasq
It was that simple! Now, the NetworkManager service was being instructed to use Dnsmasq to resolve all domains, instead of the native system-resolve service. However, this configuration alone was insufficient to route all local development domains to the localhost IP address. I still needed to add a specific Dnsmasq configuration file, to instruct it to not query the Internet’s DNS servers for requests of specific TLDs (our development TLDs).
The file that was missing needed to be inside the
dnsmasq.dfolder, which is very close to the previous folder:/etc/NetworkManager/dnsmasq.d. Creating a file there, with the namedevelopment-tld.conf, and adding a single line withaddress=/development/127.0.0.1is all that was missing. Now, Dnsmasq won’t query the Internet’s DNS servers to know what’s the IP for every domain that uses.developmentTLD.We use
.developmentas a local development TLD, but we could use the exact same steps to use any other TLD, like.testJust take note that whatever you use, won’t reach the Internet. So, if you use.com, for example, all requests for domains with that TLD will try to be resolved to your machine. Stick with a TLD that you know it’s not a “valid” one on the Internet, to avoid such problems.Finally, I had everything properly configured. I only needed to disable the system-resolve service, permanently, and restart the NetworkManager service. The commands to do so were the following:
sudo systemctl disable systemd-resolved.service sudo systemctl stop systemd-resolved.service sudo rm /etc/resolv.conf sudo systemctl restart network-manager.service
The line
sudo rm /etc/resolv.confforces the O.S. to regenerate the file with the new, updated, configuration, and the last line forces NetworkManager to load the all-new configuration.Now I had all my development domains auto-resolving to the localhost machine, even with domains that I never configured in the hosts file. Here’s an excerpt of what happens when pinging any domain with the configured
.developmentTLD:▶ ping infraspeak.development PING infraspeak.development (127.0.0.1) 56(84) bytes of data. 64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.025 ms 64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.038 ms --- infraspeak.development ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 32ms rtt min/avg/max/mdev = 0.025/0.031/0.038/0.008 ms ▶ ping non-existing-subdomain.infraspeak.development PING non-existing-subdomain.infraspeak.development (127.0.0.1) 56(84) bytes of data. 64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.027 ms 64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.041 ms --- non-existing-subdomain.infraspeak.development ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 20ms rtt min/avg/max/mdev = 0.027/0.034/0.041/0.007 ms ▶ ping customer-x.infraspeak.development PING customer-x.infraspeak.development (127.0.0.1) 56(84) bytes of data. 64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.025 ms 64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.045 ms --- customer-x.infraspeak.development ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 23ms rtt min/avg/max/mdev = 0.025/0.035/0.045/0.010 ms
Pretty sweet, right? No more messing around the hosts file!
-
This article was originally submitted on the Infraspeak tech blog.
Infraspeak is a startup founded in 2015, focused on developing the best maintenance management software in the market. Built around simplicity and a user-friendly UI/UX, we have more than 25000 buildings being managed every day, with the help of our product.
You might think that due to this number, there’s a big team of software engineers developing and maintaining the software. You’d be wrong. The Product Team is composed of 12 people, distributed between Backend, Frontend, Mobile, AI, and Integrations, and I’m also counting with the CTO.
Although we’re actively recruiting new developers to the team, we’re perfectly comfortable in maintaining such a user base and still making sure we keep improving the codebase with optimizations, as well as deploying new features because we automate as much as possible. One automation we have in place is the Continuous Integration pipeline.
The need for a Continuous Integration pipeline came from the fact that we were quickly becoming a critical dependency for our customers daily operations and, because of that, we needed to focus on constant quality assurance of our work. We needed a way of automating the integration testing of the work that was merged to the master branch (which is then deployed to production) with as minimal human interference as possible.
When searching for options for a pipeline system, we were looking for something simple and quick to implement, but also easy to maintain. We had three ways of doing this:
- Invest in preparing and actively maintaining a dedicated infrastructure for a self-hosted system, like Jenkins or Drone CI.
- Pay a lot of money, recurrently, to use another external tool, like Travis CI or Circle CI.
- Use Bitbucket Pipelines, which is natively integrated with their software repository, which we were already using.
Since we have a relatively small team, allocating resources to configure a dedicated CI infrastructure, and actively support it, was very hard to reason about, so this option was quickly disregarded. We looked at Travis CI and Circle CI with good eyes, because we didn’t have to handle the maintenance ourselves, but even being a well-funded company, their pricing plans were very expensive for our first attempt at using a pipeline system. We needed to gather more information, and experience, before requesting more resources to be invested in this.
In the end, we went with Bitbucket Pipelines. Since we were already a paid customer, we had access to a 500 minutes pipeline execution plan without any additional costs. For what we wanted to do at the moment, it was good enough.
Activating Bitbucket Pipelines was as simple as clicking on the dedicated “Enable Pipelines” button, available in the repository settings page. The interesting and a little bit more complicated part was creating the configuration YAML file according to the project specification so that Bitbucket knew when to run and how to instantiate and automatically configure the pipeline’s infrastructure. The main goal was to activate the pipeline in specific points on our branching strategy.
We have a pretty standard branching strategy: we have a master branch, always in sync with production, we have a development branch and we have n working branches corresponding to an active task. As soon as developers are finished with their task, they send a PR targeting the development branch, which then is peer-reviewed and, finally, merged. At the end of the sprint cycle, the development branch gets merged to master and deployed to production.
To be confident about the changes we continuously made to the project, and to minimize the need for manual testing between merges to the development and the master branch, without sacrificing speed and agility of development, we needed the pipeline to run on key events of our workflow:
- When a PR is opened targeting the master or the development branch.
- When a push is done directly to the development or the master branch.
Because we had a limited execution time, we couldn’t include every push to the working branches but assumed that each developer would run the full test suite locally and only push the changes when they had them all passing. The pipeline would only continuously check the integration of all developers code into the development and the master branch, as we knew that those were the points in time where bugs, conflicts, and other problems could happen.
We decided that the first project which we were to have the pipeline configured to test was our main API layer. It’s where we have the main business logic and data, making it one of the main critical points of failure to our business. If it failed, then all other tools we provide were going to fail too. Having this in mind, we started to map all the steps required to automatically instantiate the project and run the full test suite (unit and feature tests):
- Checkout the project from the repository.
- Install all the dependencies and instantiate a PostgreSQL database.
- Run the full test suite.
Digging through the Bitbucket Pipelines documentation, we learned that it’s all run inside a dockerized environment and that the main base image they recommend using would check out the correct branch and make the code available to all other steps of the pipeline. That was exactly what we needed to handle the first step.
For step two, we needed to have a way to run Composer, to install the dependencies, and run PHP for the test suite. The Bitbucket Pipelines documentation states that each pipeline step can have a dedicated docker image running. That would support our initial idea for running Composer and the PHPUnit test suite in two different steps.
The documentation also refers to the fact that we can have shared containers (they call it services) running and accessible from all steps throughout the whole pipeline execution. Since we rely on PostgreSQL as our persistent layer, and we had a lot of feature tests that would use that layer, it was a very much appreciated functionality that we would definitely need.
After having the first draft of the pipeline configuration file, we created a test branch, pushed the file to the repository and watched the pipeline come to life. It started to run and we were watching happily, for about a minute, then it failed hard. The logs stated that the vendor folder, which is where Composer downloads and installs the dependencies, was nowhere to be found. It seemed that it was being removed at the end of the step (at the teardown phase).
We got to scratch our heads, a lot, about this. We were installing the dependencies in the previous step and we were declaring the composer cache strategy, which is natively supported by Bitbucket and is specifically built for keeping Composer dependencies for the next step. After a little investigation, and reading a lot of documentation, we found out that this was happening because the artifacts configuration key was missing. This configuration maps the Composer vendor folder, and all files it contains, to transition to the next step.
After updating the configuration file, we got it running successfully. The vendor folder was transitioning correctly to the next step and the test suite was passing. Everything was green and we had our first successful pipeline execution!
Since it was running, it was time to optimize it. The first optimization was around the Composer dependencies install. It was slow, taking around three minutes to install all the dependencies. Was there something we could do to improve this? After a little more digging, we found out that we should use a caching strategy. Composer, when running locally in your machine, saves a reference to the remote dependencies repositories, allowing it to skip several steps when fetching those dependencies.
Since the Composer binary, on our pipeline, was running in a docker container, the references cache were compiled but they would never persist. Docker containers don’t persist data after being destroyed, unless you add volumes to them. And that’s what we needed to add: a cache volume to persist those references.
Doing that was not an easy task. The Bitbucket Pipelines documentation states that it has a predefined caching strategy for Composer, which is awesome and would save a lot of time by not requiring us to configure a personalized one, but it forgot to mention that if you use the default Composer docker image you need to also declare the /tmp folder as the one that needs to be cached.
But after we finished updating the configuration file, the time it took to install the dependencies went down to around fifteen seconds. That’s fast! Considering that this was intended to run several times, per developer, in a normal workday, it would stretch out our available execution time cap.
And it was pretty much it. We only needed to make it run on the predefined events (pull requests and pushes to master or development branch) and we had our pipeline fully working so we made the PR to the development branch. Our final configuration file was similar to the following:
image: atlassian/default-image:latest pipelines: pull-requests: '**': - step: name: Install dependencies image: composer caches: - composer artifacts: - vendor/** script: - composer install --ignore-platform-reqs - step: name: Run tests image: php:7.2-cli script: - ln -f -s .env.pipeline .env - php artisan key:generate - php artisan migrate - php artisan passport:keys - printf "\n" | php artisan passport:client --password - vendor/bin/phpunit -c tests/phpunit.xml --testsuite Unit --no-coverage - vendor/bin/phpunit -c tests/phpunit.xml --testsuite Feature --no-coverage - vendor/bin/phpunit -c tests/phpunit.xml --testsuite Domain --no-coverage services: - postgres branches: '{master,dev-sprint-*}': - step: name: Install dependencies image: composer caches: - composer artifacts: - vendor/** script: - composer install --ignore-platform-reqs - step: name: Run tests image: php:7.2-cli script: - ln -f -s .env.pipeline .env - php artisan key:generate - php artisan migrate - php artisan passport:keys - printf "\n" | php artisan passport:client --password - vendor/bin/phpunit -c tests/phpunit.xml --testsuite Unit --no-coverage - vendor/bin/phpunit -c tests/phpunit.xml --testsuite Feature --no-coverage - vendor/bin/phpunit -c tests/phpunit.xml --testsuite Domain --no-coverage services: - postgres definitions: caches: composer: /tmp services: postgres: image: postgres:10.5 environment: POSTGRES_DB: database POSTGRES_USER: root POSTGRES_PASSWORD: rootWith Bitbucket Pipelines handling our continuous integration testing workflow, every time some sneaky bug tried to enter our codebase, the pipeline would fail, informing the author of the commit via e-mail, warning to check the pipeline logs and apply the necessary correction to the code submitted.
As long as we continued to add tests for every new feature, bug fix, and improvement, the pipeline would take care of checking every test case for possible problems. This had a very positive impact in our team workflow and improved our code quality because lesser bugs were merged into the codebase. It helped the team shift a little bit more from a reactive to a more preventive position. It’s better for the pipeline to catch the bugs and break, than our customers’ catch the bugs, lose work and then lose those clients!