A Guide to NDAs for Startups and Entrepreneurs

Nobody wants to sign your NDA. VCs, software engineers, freelancers — will all likely throw some shade your way when you bring up “non-disclosure agreement.” But if you’re an entrepreneur, you still need to protect your confidential information in certain circumstances.

In this post, we will cover why you need an NDA, who you should expect to use one with, what should be in it, how to draft the document, the proper timing for signature requests, and how to enforce a non-disclosure agreement.

Here we go.

What is an NDA?

NDA Signature

An NDA is a legal document that is intended to set up a confidential relationship between two or more parties, made up of a party disclosing information and a party receiving information. The non-disclosure agreement stipulates that information shared between parties should be used only for the purpose of that specific partnership, so as to protect the market position and the competitive advantage of the disclosing party.

Basically, the two parties agree to share information in order to help each other — while promising not to use that information outside of the relationship in such a way that could damage the other party’s interests.

NDAs can be written as a section of an employment contract or separately drawn up. A non-disclosure agreement can also be referred to as: Confidentiality Agreement, a Confidential Disclosure Agreement, a Proprietary Information Agreement, a Secrecy Agreement, a Proprietary Information and Inventions Agreement, or for that matter, any other arrangement of words indicating confidentiality that a disclosing party might prefer.

Why Do You Need an NDA?

If you are building a business, presumably there is certain information you wouldn’t want getting into the hands of potential competitors. Here are some of the main assets businesses seek to protect through an NDA.

Intellectual Property

Because of the incremental but speedy nature of software development, Intellectual Property Rights (IPR) patents are quickly becoming irrelevant in the industry. Most developers working on new projects are free to use code from open source libraries like ReactJS or AngularJS, which means the wheel is hardly ever reinvented for new software.

In most cases, it would be counterproductive to file a patent for code that would be outdated by the time said patent was delivered. Instead, companies can protect their software innovation by including a work-for-hire clause that requires that the rights to the code and products developed by employees and freelancers are automatically transferred to the company.

This can be included as a clause within the NDA, or for those who are really concerned their pot of gold will be pilfered, a separate Intellectual Property Rights Agreement like this one can be used. However, this may feel excessive to some and can be taken as an insult to one’s professional integrity. In fact, legal experts suggest that common law will usually be enough to uphold an employer’s rights to all of the creative works developed by employees in their service. However, this is not true of independent contractors, who will retain rights to their work unless otherwise agreed upon. Therefore, entrepreneurs should take care to include a work-for-hire or transfer of rights clause in an NDA or contract when working with freelancers.

Proprietary Information and Trade Secrets

There is a slight distinction between proprietary information and trade secrets, and a business may reasonably want to protect both. Proprietary information is any unique information that a business uses to operate, including:

  • Suppliers
  • Manufacturing agreements
  • Marketing strategy
  • Development processes
  • Pricing
  • Customer lists/client info
  • Research & data
  • Formulas and algorithms
  • Unique code
  • Test results
  • Product development plans

The list goes on. Not all proprietary information is, or needs to be, confidential. The business must decide what proprietary information they want to mark as confidential, thereby making it a trade secret. The Uniform Trade Secrets Act defines a trade secret as:

(i) information that derives independent economic value, actual or potential, from not being generally known to, and not being readily ascertainable by proper means by, other persons who can obtain economic value from its disclosure or use, and (ii) is the subject of efforts that are reasonable under the circumstances to maintain its secrecy.

Basically, this is your “secret sauce.” Information that helps you make a profit and is not readily available to others — the stuff that you wouldn’t want your competitors to have.

The most important reason for using an NDA is to guard against future legal expenses. You can hopefully avoid future litigation by protecting your trade secrets and insisting on ownership of the intellectual property developed at your business.

The NDA should be designed to provide a measure of protection against having to sue someone who has improperly used your information and, likewise, avert the risk of being sued for rights or royalties by a former employee or freelancer who wants to claim rights to products they helped develop.

What’s in an NDA?

This section will go over the eight clauses of an NDA, two of which are optional and should be carefully considered if written in.

Definition and Scope
Non-use
Non-compete
Representative Provision
Duration
Return Clause
Arbitration Clause
Attorney Provision Fee

Sample NDA

1. Definition and Scope

The definition and scope of what will be considered confidential are the most important parts of the NDA. It’s best to be specific when designating what proprietary information will be a trade secret. Without further clarification, terms like “proprietary information” or “business practices” are vague, and will likely not be enforceable if put to a legal test. Plus, under such broad terms, all information exchanged could be considered confidential. In a one-way NDA, the receiving party would be wise to walk away.

Whether it be your codebase, algorithms, client lists, product roadmap, or any other valuable proprietary information, it should be specified as confidential in the NDA. Future communications or documentation including these confidential items should also be marked as such with a “confidential footer.”

2. Non-use

As we’ve mentioned in an earlier section of this article, the whole purpose of an NDA is to formally agree that the information exchanged is only to be used for the parties entering into partnership. Therefore, your NDA will need to have a non-use clause. In addition to agreeing not to disclose your confidential information, the non-use clause also prohibits the receiving party from making use of the information in such a way that would be damaging to the vital business interests of the disclosing party.

The non-use clause is intended to prevent:

  • Formation of new businesses in direct competition to the disclosing party
  • Receiving parties from using confidential proprietary information as a bargaining chip for personal gain or new job opportunities
  • Existing competitors from soliciting current employees or freelancers for the knowledge they have of your business secrets

The non-use clause should not prevent:

  • The receiving party from using new skills learned in the future
  • The receiving party from working on projects with similar but technically different applications in the future

🌟 To make the non-use clause more acceptable to the receiving party, consider making the restriction only as long as the duration of the project or employment contract

3. Non-compete

The jury is out on whether or not it’s appropriate to use a non-compete clause in your NDA. Any developer worth their salt is probably not going to sign anything that restricts their future employment options.

Freelancers will be especially sensitive to any non-compete language. If you must have a non-compete clause, it should only be designed to prevent the employee or freelance developer from taking your technology and business model to a direct competitor or soliciting your employees to start a competing business venture. It typically should notprevent them from working in the same industry or location, for any period of time.

In truth, both of these objectives can be achieved with the non-use clause. Another reason to tread lightly when it comes to non-compete is that many state courts deem non-compete restrictions a barrier to free trade and seldom enforce this part of an NDA anyway.

4. Representative Provision

To provide your partners with more flexibility and freedom to complete the project as they see fit, you should consider including a representative provision. The representative provision permits confidential information to be shared with associates of the receiving party for the purpose of completing the project. Once information is shared with a representative, he or she will be bound by terms of the NDA as well.

This clause should define who may be considered a “representative,” and can also require that the recipient party inform the disclosing party of any additional associates who will be collaborating on the project.

5. Duration

When drawing up an NDA, you will want to define a reasonable length of time for the agreement to remain in force. Too short, and you expose yourself to the risk of competitors learning your trade secrets before you’re able to establish a firm competitive edge. Too long, and skilled developers will not want to work with you. Most experts suggest anywhere from two to five years as a fair term of obligation, after which the agreement will automatically terminate.

6. Return Clause

Throughout the course of a project, you will have likely transferred a lot of files, instructions, data, communications, and other materials containing your business’s confidential information. Upon termination or completion of the project, you will want the return or destruction of the most sensitive of those documents. This clause stipulates that upon the written request of the disclosing party, the recipient of confidential information returns or destroys the material in their possession.

Just like confidential information should be specifically defined, what must be returned or destroyed should also be specified. Unless you are a government defense contractor, it won’t be necessary to ask someone to delete every single email or file they have received from you — just protect the key ingredients of your secret sauce!

7. Arbitration Clause

Should things go awry, having a mutual understanding of how to handle a dispute can save you money and hassle. Lawyering-up and going to court to file a lawsuit is very, very expensive. This should be considered a last resort. the arbitration clause is the place to outline alternatives to official litigation.

Your preferred path should be good faith mediation between parties. This is essentially a discussion facilitated by a moderator to air grievances and explore solutions. If informal mediation fails to resolve differences, you can opt for a binding (or non-binding) arbitration committee. This like a mini-trial outside of the official judicial system. Arbitration committees can deliver decisions, which are typically a good indicator of how an actual court proceeding would go. Your last and most resource draining option is to bring your dispute to trial.

Lastly, your arbitration clause should identify the territory whose laws will be used to govern the agreement, including any disputes.

8. Attorney Provision Fee

An attorney provision fee requires that the “losing” party of a lawsuit pay the legal fees of the prevailing party. Of course, this will help to ease the burden of litigation, but it also conflicts with mediation and arbitration for that reason. When passions are inflamed and both sides believe they are in the right, removing the threat of financial punishment will mean that parties are less likely to cooperate. With that in mind, you may choose to include or not include an attorney provision fee in your NDA.

 

What’s Not in an NDA?

What's not in an NDA?

Public knowledge. As mentioned earlier in this article, most software applications today are developed using snippets of open source code. If the codebase used to develop your project is publically available, it cannot be included as part of your confidential information. That being said, new changes made to source code may become part of your business advantage, and can be protected under the NDA.

Prior knowledge and independently developed knowledge. Rights to knowledge or innovations credited to the freelancer prior to the partnership will be retained by the freelancer developer. The same goes for knowledge developed independently of the project, even if it occurred during the time of the partnership. Freelancers would be wise make an itemized list of any valuable knowledge or previous inventions to be covered under this clause.

Third Party Information. Freelance software developers often work with various clients, and they may have more than one non-disclosure agreement in force concurrently. The NDA should specifically note that information obtained by a third party is not confidential. This protects the freelancer from being unfairly sued for breach of contract in the case that information disclosed from various parties overlaps.

 

Now you know what is and isn’t in an NDA, who should you get to sign your NDA and when? Should you ask for signatures from freelancers or VCs? When should you lawyer up and how does one enforce an NDA? Make sure to read the Guide to NDAs for Startups and Entrepreneurs for answers.

Cloud Computing Journey – Part 2

Cloud service provider questionnaire

In my previous post I gave you a short introduction to cloud computing.

When engaging with cloud service provider, it is important to evaluate the provider’s maturity level by asking the provider, as many questions as possible to allow you the comfort level to sign a contract.

Below is a sample questionnaire I recommend you to ask the cloud service provider.

Privacy related questions:

  • Does the cloud service provider has an official privacy policy?
  • Where are the cloud service provider data centers located around the world?
  • Are the cloud service provider data centers compliant with the EU Directive 95/46/EC?
  • Are the cloud service provider data centers compliant with the General Data Protection Regulation (GDPR)?

 

Availability related questions:

  • What is the SLA of the cloud service provider? (Please elaborate)
  • Does the cloud service provider publish information about system issues or outages?
  • What compensation does the cloud service provider offer in case of potential financial loss due to lack of availability?
  • Does the cloud service provider sync data between more than one data center on the same region?
  • How many data centers does the cloud service provider has in the same region?
  • Does the cloud service provider have business continuity processes and procedures? (Please elaborate)
    • What is the cloud service provider’s RTO?
    • What is the cloud service provider’s RPO?
  • What is the cloud service provider disaster recovery strategy?
  • Does the cloud service provider have change management processes? (Please elaborate)
  • Does the cloud service provider have backup processes? (Please elaborate)

 

Interoperability related questions:

  • Does the cloud service provider support security event monitoring using an API? (Please elaborate)
  • Does the cloud service provider support infrastructure related event monitoring using an API? (Please elaborate)

 

Security related questions:

  • What is the cloud service provider’s audit trail process for my organizational data stored or processed? (Please elaborate)
  • What logical controls does the cloud service provider use for my organizational data stored or processed? (Please elaborate)
  • What physical controls does the cloud service provider use for my organizational data stored or processed? (Please elaborate)
  • Does the cloud service provider encrypt data at transit? (Please elaborate)
  • Does the cloud service provider encrypt data at rest? (Please elaborate)
    • What encryption algorithm is been used?
    • What encryption key size is been used?
    • Where are the encryption keys stored?
    • At what interval does the cloud service provider rotate the encryption keys?
    • Does the cloud service provider support BYOK (Bring your own keys)?
    • Does the cloud service provider support HYOK (Hold your own keys):
    • At what level does the data at rest been encrypted? (Storage, database, application, etc.)
  • What security controls are been used by the cloud service provider to protect the cloud service itself?
  • Is there an on-going process for Firewall rule review been done by the cloud service provider? (Please elaborate)
  • Are all cloud service provider’s platform (Operating system, database, middleware, etc.) been hardened according to best practices? (Please elaborate)
  • Does the cloud service provider perform an on-going patch management process for all hardware and software? (Please elaborate)
  • What security controls are been used by the cloud service provider to protect against data leakage in a multi-tenant environment?
  • How does the cloud service provider perform access management process? (Please elaborate)
  • Does the cloud service provider enforce 2-factor authentication for accessing all management interfaces?
  • Is the authentication to the cloud service based on standard protocols such as SAML, OAuth, OpenID?
  • How many employees at the cloud service provider will have access to my organizational data? (Infrastructure and database level)
  • Is there an access to the cloud service provider’s 3rd party suppliers to my organizational data?
  • Does the cloud service provider enforce separation between production and development/test environments? (Please elaborate)
  • What is the cloud service provider’s password policy (Operating system, database, network components, etc.) for systems that store or process my organizational data?
  • Is it possible to schedule security survey and penetration test on the systems that stored my organizational data?
  • Does the cloud service provider have incident response processes and procedures? (Please elaborate)
  • What are the escalation processes in case of security incident related to my organizational data? (Please elaborate)
  • What are the cloud service provider’s processes and controls against distributed denial-of-service? (Please elaborate)
  • Does the cloud service provider have vulnerability management processes? (Please elaborate)
  • Does the cloud service provider have secure development lifecycle (SDLC) process? (Please elaborate)

 

Compliance related questions:

  • Is the cloud service provider compliant with certifications or standards? (Please elaborate)
  • What is the level of compliance with the Cloud Security Alliance Matrix (https://cloudsecurityalliance.org/research/ccm)?
  • Is it possible to receive a copy of internal audit report performed on the cloud service in the last 12 months?
  • Is it possible to receive a copy of external audit report performed on the cloud service in the last 12 months?
  • Is it possible to perform an on site audit on the cloud service provider’s data center and activity?

 

Contract termination related questions:

  • What are the cloud service provider’s contract termination options?
  • What options does the cloud service provider allow me to export my organizational data stored on the cloud?
  • Is there a process for data deletion in case of contract termination?
  • What standard does the cloud service provider use for data deletion?

 

Stay tuned for my next article.

 

Here are some recommended articles:

Separation Anxiety: A Tutorial for Isolating Your System with Linux Namespaces

With the advent of tools like Docker, Linux Containers, and others, it has become super easy to isolate Linux processes into their own little system environments. This makes it possible to run a whole range of applications on a single real Linux machine and ensure no two of them can interfere with each other, without having to resort to using virtual machines. These tools have been a huge boon to PaaS providers. But what exactly happens under the hood?

These tools rely on a number of features and components of the Linux kernel. Some of these features were introduced fairly recently, while others still require you to patch the kernel itself. But one of the key components, using Linux namespaces, has been a feature of Linux since version 2.6.24 was released in 2008.

Anyone familiar with chroot already has a basic idea of what Linux namespaces can do and how to use namespace generally. Just as chroot allows processes to see any arbitrary directory as the root of the system (independent of the rest of the processes), Linux namespaces allow other aspects of the operating system to be independently modified as well. This includes the process tree, networking interfaces, mount points, inter-process communication resources and more.

Why Use Namespaces for Process Isolation?

In a single-user computer, a single system environment may be fine. But on a server, where you want to run multiple services, it is essential to security and stability that the services are as isolated from each other as possible. Imagine a server running multiple services, one of which gets compromised by an intruder. In such a case, the intruder may be able to exploit that service and work his way to the other services, and may even be able compromise the entire server. Namespace isolation can provide a secure environment to eliminate this risk.

For example, using namespacing, it is possible to safely execute arbitrary or unknown programs on your server. Recently, there has been a growing number of programming contest and “hackathon” platforms, such as HackerRank, TopCoder, Codeforces, and many more. A lot of them utilize automated pipelines to run and validate programs that are submitted by the contestants. It is often impossible to know in advance the true nature of contestants’ programs, and some may even contain malicious elements. By running these programs namespaced in complete isolation from the rest of the system, the software can be tested and validated without putting the rest of the machine at risk. Similarly, online continuous integration services, such as Drone.io, automatically fetch your code repository and execute the test scripts on their own servers. Again, namespace isolation is what makes it possible to provide these services safely.

Namespacing tools like Docker also allow better control over processes’ use of system resources, making such tools extremely popular for use by PaaS providers. Services like Heroku and Google App Engine use such tools to isolate and run multiple web server applications on the same real hardware. These tools allow them to run each application (which may have been deployed by any of a number of different users) without worrying about one of them using too many system resources, or interfering and/or conflicting with other deployed services on the same machine. With such process isolation, it is even possible to have entirely different stacks of dependency softwares (and versions) for each isolated environment!

If you’ve used tools like Docker, you already know that these tools are capable of isolating processes in small “containers”. Running processes in Docker containers is like running them in virtual machines, only these containers are significantly lighter than virtual machines. A virtual machine typically emulates a hardware layer on top of your operating system, and then runs another operating system on top of that. This allows you to run processes inside a virtual machine, in complete isolation from your real operating system. But virtual machines are heavy! Docker containers, on the other hand, use some key features of your real operating system, including namespaces, and ensure a similar level of isolation, but without emulating the hardware and running yet another operating system on the same machine. This makes them very lightweight.

Process Namespace

Historically, the Linux kernel has maintained a single process tree. The tree contains a reference to every process currently running in a parent-child hierarchy. A process, given it has sufficient privileges and satisfies certain conditions, can inspect another process by attaching a tracer to it or may even be able to kill it.

With the introduction of Linux namespaces, it became possible to have multiple “nested” process trees. Each process tree can have an entirely isolated set of processes. This can ensure that processes belonging to one process tree cannot inspect or kill – in fact cannot even know of the existence of – processes in other sibling or parent process trees.

Every time a computer with Linux boots up, it starts with just one process, with process identifier (PID) 1. This process is the root of the process tree, and it initiates the rest of the system by performing the appropriate maintenance work and starting the correct daemons/services. All the other processes start below this process in the tree. The PID namespace allows one to spin off a new tree, with its own PID 1 process. The process that does this remains in the parent namespace, in the original tree, but makes the child the root of its own process tree.

With PID namespace isolation, processes in the child namespace have no way of knowing of the parent process’s existence. However, processes in the parent namespace have a complete view of processes in the child namespace, as if they were any other process in the parent namespace.

 

It is possible to create a nested set of child namespaces: one process starts a child process in a new PID namespace, and that child process spawns yet another process in a new PID namespace, and so on.

With the introduction of PID namespaces, a single process can now have multiple PIDs associated with it, one for each namespace it falls under. In the Linux source code, we can see that a struct named pid, which used to keep track of just a single PID, now tracks multiple PIDs through the use of a struct named upid:

struct upid {
  int nr;                     // the PID value
  struct pid_namespace *ns;   // namespace where this PID is relevant
  // ...
};

struct pid {
  // ...
  int level;                  // number of upids
  struct upid numbers[0];     // array of upids
};

To create a new PID namespace, one must call the clone() system call with a special flag CLONE_NEWPID. (C provides a wrapper to expose this system call, and so do many other popular languages.) Whereas the other namespaces discussed below can also be created using the unshare() system call, a PID namespace can only be created at the time a new process is spawned using clone(). Once clone() is called with this flag, the new process immediately starts in a new PID namespace, under a new process tree. This can be demonstrated with a simple C program:

#define _GNU_SOURCE
#include <sched.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/wait.h>
#include <unistd.h>

static char child_stack[1048576];

static int child_fn() {
  printf("PID: %ld\n", (long)getpid());
  return 0;
}

int main() {
  pid_t child_pid = clone(child_fn, child_stack+1048576, CLONE_NEWPID | SIGCHLD, NULL);
  printf("clone() = %ld\n", (long)child_pid);

  waitpid(child_pid, NULL, 0);
  return 0;
}

Compile and run this program with root privileges and you will notice an output that resembles this:

clone() = 5304
PID: 1

The PID, as printed from within the child_fn, will be 1.

Even though this namespace tutorial code above is not much longer than “Hello, world” in some languages, a lot has happened behind the scenes. The clone() function, as you would expect, has created a new process by cloning the current one and started execution at the beginning of the child_fn() function. However, while doing so, it detached the new process from the original process tree and created a separate process tree for the new process.

Try replacing the static int child_fn() function with the following, to print the parent PID from the isolated process’s perspective:

static int child_fn() {
  printf("Parent PID: %ld\n", (long)getppid());
  return 0;
}

Running the program this time yields the following output:

clone() = 11449
Parent PID: 0

Notice how the parent PID from the isolated process’s perspective is 0, indicating no parent. Try running the same program again, but this time, remove the CLONE_NEWPID flag from within the clone() function call:

pid_t child_pid = clone(child_fn, child_stack+1048576, SIGCHLD, NULL);

This time, you will notice that the parent PID is no longer 0:

clone() = 11561
Parent PID: 11560

However, this is just the first step in our tutorial. These processes still have unrestricted access to other common or shared resources. For example, the networking interface: if the child process created above were to listen on port 80, it would prevent every other process on the system from being able to listen on it.

Linux Network Namespace

This is where a network namespace becomes useful. A network namespace allows each of these processes to see an entirely different set of networking interfaces. Even the loopback interface is different for each network namespace.

Isolating a process into its own network namespace involves introducing another flag to the clone() function call: CLONE_NEWNET;

#define _GNU_SOURCE
#include <sched.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/wait.h>
#include <unistd.h>


static char child_stack[1048576];

static int child_fn() {
  printf("New `net` Namespace:\n");
  system("ip link");
  printf("\n\n");
  return 0;
}

int main() {
  printf("Original `net` Namespace:\n");
  system("ip link");
  printf("\n\n");

  pid_t child_pid = clone(child_fn, child_stack+1048576, CLONE_NEWPID | CLONE_NEWNET | SIGCHLD, NULL);

  waitpid(child_pid, NULL, 0);
  return 0;
}

Output:

Original `net` Namespace:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 00:24:8c:a1:ac:e7 brd ff:ff:ff:ff:ff:ff


New `net` Namespace:
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN mode DEFAULT group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

What’s going on here? The physical ethernet device enp4s0 belongs to the global network namespace, as indicated by the “ip” tool run from this namespace. However, the physical interface is not available in the new network namespace. Moreover, the loopback device is active in the original network namespace, but is “down” in the child network namespace.

In order to provide a usable network interface in the child namespace, it is necessary to set up additional “virtual” network interfaces which span multiple namespaces. Once that is done, it is then possible to create Ethernet bridges, and even route packets between the namespaces. Finally, to make the whole thing work, a “routing process” must be running in the global network namespace to receive traffic from the physical interface, and route it through the appropriate virtual interfaces to to the correct child network namespaces. Maybe you can see why tools like Docker, which do all this heavy lifting for you, are so popular!

 

To do this by hand, you can create a pair of virtual Ethernet connections between a parent and a child namespace by running a single command from the parent namespace:

ip link add name veth0 type veth peer name veth1 netns <pid>

Here, <pid> should be replaced by the process ID of the process in the child namespace as observed by the parent. Running this command establishes a pipe-like connection between these two namespaces. The parent namespace retains the veth0 device, and passes the veth1 device to the child namespace. Anything that enters one of the ends, comes out through the other end, just as you would expect from a real Ethernet connection between two real nodes. Accordingly, both sides of this virtual Ethernet connection must be assigned IP addresses.

Mount Namespace

Linux also maintains a data structure for all the mountpoints of the system. It includes information like what disk partitions are mounted, where they are mounted, whether they are readonly, et cetera. With Linux namespaces, one can have this data structure cloned, so that processes under different namespaces can change the mountpoints without affecting each other.

Creating separate mount namespace has an effect similar to doing a chroot(). chroot() is good, but it does not provide complete isolation, and its effects are restricted to the root mountpoint only. Creating a separate mount namespace allows each of these isolated processes to have a completely different view of the entire system’s mountpoint structure from the original one. This allows you to have a different root for each isolated process, as well as other mountpoints that are specific to those processes. Used with care per this tutorial, you can avoid exposing any information about the underlying system.

 

The clone() flag required to achieve this is CLONE_NEWNS:

clone(child_fn, child_stack+1048576, CLONE_NEWPID | CLONE_NEWNET | CLONE_NEWNS | SIGCHLD, NULL)

Initially, the child process sees the exact same mountpoints as its parent process would. However, being under a new mount namespace, the child process can mount or unmount whatever endpoints it wants to, and the change will affect neither its parent’s namespace, nor any other mount namespace in the entire system. For example, if the parent process has a particular disk partition mounted at root, the isolated process will see the exact same disk partition mounted at the root in the beginning. But the benefit of isolating the mount namespace is apparent when the isolated process tries to change the root partition to something else, as the change will only affect the isolated mount namespace.

Interestingly, this actually makes it a bad idea to spawn the target child process directly with the CLONE_NEWNS flag. A better approach is to start a special “init” process with the CLONE_NEWNS flag, have that “init” process change the “/”, “/proc”, “/dev” or other mountpoints as desired, and then start the target process. This is discussed in a little more detail near the end of this namespace tutorial.

Other Namespaces

There are other namespaces that these processes can be isolated into, namely user, IPC, and UTS. The user namespace allows a process to have root privileges within the namespace, without giving it that access to processes outside of the namespace. Isolating a process by the IPC namespace gives it its own interprocess communication resources, for example, System V IPC and POSIX messages. The UTS namespace isolates two specific identifiers of the system: nodename and domainname.

A quick example to show how UTS namespace is isolated is shown below:

#define _GNU_SOURCE
#include <sched.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/utsname.h>
#include <sys/wait.h>
#include <unistd.h>


static char child_stack[1048576];

static void print_nodename() {
  struct utsname utsname;
  uname(&utsname);
  printf("%s\n", utsname.nodename);
}

static int child_fn() {
  printf("New UTS namespace nodename: ");
  print_nodename();

  printf("Changing nodename inside new UTS namespace\n");
  sethostname("GLaDOS", 6);

  printf("New UTS namespace nodename: ");
  print_nodename();
  return 0;
}

int main() {
  printf("Original UTS namespace nodename: ");
  print_nodename();

  pid_t child_pid = clone(child_fn, child_stack+1048576, CLONE_NEWUTS | SIGCHLD, NULL);

  sleep(1);

  printf("Original UTS namespace nodename: ");
  print_nodename();

  waitpid(child_pid, NULL, 0);

  return 0;
}

This program yields the following output:

Original UTS namespace nodename: XT
New UTS namespace nodename: XT
Changing nodename inside new UTS namespace
New UTS namespace nodename: GLaDOS
Original UTS namespace nodename: XT

Here, child_fn() prints the nodename, changes it to something else, and prints it again. Naturally, the change happens only inside the new UTS namespace.

More information on what all of the namespaces provide and isolate can be found in the tutorial here

Cross-Namespace Communication

Often it is necessary to establish some sort of communication between the parent and the child namespace. This might be for doing configuration work within an isolated environment, or it can simply be to retain the ability to peek into the condition of that environment from outside. One way of doing that is to keep an SSH daemon running within that environment. You can have a separate SSH daemon inside each network namespace. However, having multiple SSH daemons running uses a lot of valuable resources like memory. This is where having a special “init” process proves to be a good idea again.

The “init” process can establish a communication channel between the parent namespace and the child namespace. This channel can be based on UNIX sockets or can even use TCP. To create a UNIX socket that spans two different mount namespaces, you need to first create the child process, then create the UNIX socket, and then isolate the child into a separate mount namespace. But how can we create the process first, and isolate it later? Linux provides unshare(). This special system call allows a process to isolate itself from the original namespace, instead of having the parent isolate the child in the first place. For example, the following code has the exact same effect as the code previously mentioned in the network namespace section:

#define _GNU_SOURCE
#include <sched.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/wait.h>
#include <unistd.h>


static char child_stack[1048576];

static int child_fn() {
  // calling unshare() from inside the init process lets you create a new namespace after a new process has been spawned
  unshare(CLONE_NEWNET);

  printf("New `net` Namespace:\n");
  system("ip link");
  printf("\n\n");
  return 0;
}

int main() {
  printf("Original `net` Namespace:\n");
  system("ip link");
  printf("\n\n");

  pid_t child_pid = clone(child_fn, child_stack+1048576, CLONE_NEWPID | SIGCHLD, NULL);

  waitpid(child_pid, NULL, 0);
  return 0;
}

And since the “init” process is something you have devised, you can make it do all the necessary work first, and then isolate itself from the rest of the system before executing the target child.

Conclusion

This tutorial is just an overview of how to use namespaces in Linux. It should give you a basic idea of how a Linux developer might start to implement system isolation, an integral part of the architecture of tools like Docker or Linux Containers. In most cases, it would be best to simply use one of these existing tools, which are already well-known and tested. But in some cases, it might make sense to have your very own, customized process isolation mechanism, and in that case, this namespace tutorial will help you out tremendously.

There is a lot more going on under the hood than I’ve covered in this article, and there are more ways you might want to limit your target processes for added safety and isolation. But, hopefully, this can serve as a useful starting point for someone who is interested in knowing more about how namespace isolation with Linux really works.

Originally from Toptal

Cloud Computing Journey – Part 1

So, you decided to migrate a system to the cloud. It may be business or IT initiative, but what does it really mean switching between on premise and the cloud?

For ages, we used to manage our IT infrastructure by ourselves, on our own data centers (or network communication cabinets, for small companies…), using our purchased hardware, while maintaining and troubleshooting every software/hardware/network problem.

In the cloud, things change. In the cloud, we are one of the many customers sharing compute resources in multi-tenant environment. We have no control of the hardware or the chosen platform technology (from the servers’ hardware vendor to the storage vendor), we barely control the virtualization layer, and don’t even get me started talking about troubleshooting the network layer.

There are 3 cloud service models:

  • IaaS (Infrastructure as a service) – In this service model, the customer controls (almost) everything from the virtual servers operating system, through the application layer, up until the data itself.
  • PaaS (Platform as a service) – In this service model, the customer controls the application layer, up until the data itself.
  • SaaS (Software as a service) – In this service model, the customer has access to a close application, but the customer is the data owner and can control permissions (and auditing) of the data.

 

Once we understood those basic rules, let’s analyze what does it really means migrating to the cloud.

The word that differentiates a mature cloud service provider from a rookie is transparency.

Mature cloud service provider won’t hesitate to answer tough questions such as “Can I have a copy of your last external audit report?”, “Do you have a business continuity plan?”, “Do you have an SDLC (Software development lifecycle) process?”, etc.

When engaging with cloud service provider, it is important to know as much details about the provider as you can, until you are comfortable enough to sign a contract.

A solid contract will give you assurance about the cloud service provider’s ability to fulfill his obligations and will higher the chances of project success.

In the next couple of articles, I will try to pinpoint important tips for a successful cloud project.

Stay tuned for my next article.

Here are some recommended articles:

JSON Web Token Tutorial: An Example in Laravel and AngularJS

Getting the Most Out of Your PHP Log Files: A Practical Guide

It could rightfully be said that logs are one of the most underestimated and underutilized tools at a freelance php developer’s disposal. Despite the wealth of information they can offer, it is not uncommon for logs to be the last place a developer looks when trying to resolve a problem.

In truth, PHP log files should in many cases be the first place to look for clues when problems occur. Often, the information they contain could significantly reduce the amount of time spent pulling out your hair trying to track down a gnarly bug.

But perhaps even more importantly, with a bit of creativity and forethought, your logs files can be leveraged to serve as a valuable source of usage information and analytics. Creative use of log files can help answer questions such as: What browsers are most commonly being used to visit my site? What’s the average response time from my server? What was the percentage of requests to the site root? How has usage changed since we deployed the latest updates? And much, much more.

PHP log files

This article provides a number of tips on how to configure your log files, as well as how to process the information that they contain, in order to maximize the benefit that they provide.

Although this article focuses technically on logging for PHP developers, much of the information presented herein is fairly “technology agnostic” and is relevant to other languages and technology stacks as well.

Note: This article presumes basic familiarity with the Unix shell. For those lacking this knowledge, an Appendix is provided that introduces some of the commands needed for accessing and reading log files on a Unix system.

Our PHP Log File Example Project

As an example project for discussion purposes in this article, we will take Symfony Standard as a working project and we’ll set it up on Debian 7 Wheezy with rsyslogd, nginx, and PHP-FPM.

composer create-project symfony/framework-standard-edition my "2.6.*"

This quickly gives us a working test project with a nice UI.

Tips for Configuring Your Log Files

Here are some pointers on how to configure your log files to help maximize their value.

Error Log Confguration

Error logs represent the most basic form of logging; i.e., capturing additional information and detail when problems occur. So, in an ideal world, you would want there to be no errors and for your error logs to be empty. But when problems do occur (as they invariably do), your error logs should be one of the first stops you make on your debugging trail.

Error logs are typically quite easy to configure.

For one thing, all error and crash messages can be logged in the error log in exactly the same format in which they would otherwise be presented to a user. With some simple configuration, the end user will never need to see those ugly error traces on your site, while devops will be still able to monitor the system and review these error messages in all their gory detail. Here’s how to setup this kind of logging in PHP:

log_errors = On
error_reporting = E_ALL
error_log = /path/to/my/error/log

Another two lines that are important to include in a log file for a live site, to preclude gory levels of error detail from being to presented to users, are:

display_errors = Off
display_startup_errors = Off

System Log (syslog) Confguration

There are many generally compatible implementations of the syslog daemon in the open source world including:

  • syslogd and sysklogd – most often seen on BSD family systems, CentOS, Mac OS X, and others
  • syslog-ng – default for modern Gentoo and SuSE builds
  • rsyslogd – widely used on the Debian and Fedora families of operating systems

(Note: In this article, we’ll be using rsyslogd for our examples.)

The basic syslog configuration is generally adequate for capturing your log messages in a system-wide log file (normally /var/log/syslog; might also be /var/log/messages or /var/log/system.log depending on the distro you’re using).

The system log provides several log facilities, eight of which (LOG_LOCAL0 through LOG_LOCAL7) are reserved for user-deployed projects. Here, for example, is how you might setup LOG_LOCAL0 to write to 4 separate log files, based on logging level (i.e., error, warning, info, debug):

# /etc/rsyslog.d/my.conf

local0.err      /var/log/my/err.log
local0.warning  /var/log/my/warning.log
local0.info     -/var/log/my/info.log
local0.debug    -/var/log/my/debug.log

Now, whenever you write a log message to LOG_LOCAL0 facility, the error messages will go to /var/log/my/err.log, warning messages will go to /var/log/my/warning.log, and so on. Note, though, that the syslog daemon filters messages for each file based on the rule of “this level and higher”. So, in the example above, all error messages will appear in all four configured files, warning messages will appear in all but the error log, info messages will appear in the info and debug logs, and debug messages will only go to debug.log.

One additional important note; The - signs before the info and debug level files in the above configuration file example indicate that writes to those files should be perfomed asynchronously (since these operations are non-blocking). This is typically fine (and even recommended in most situations) for info and debug logs, but it’s best to have writes to the error log (and most prpobably the warning log as well) be synchronous.

In order to shut down a less important level of logging (e.g., on a production server), you may simply redirect related messages to /dev/null (i.e., to nowhere):

local0.debug    /dev/null # -/var/log/my/debug.log

One specific customization that is useful, especially to support some of the PHP log file parsing we’ll be discussing later in this article, is to use tab as the delimiter character in log messages. This can easily be done by adding the following file in /etc/rsyslog.d:

# /etc/rsyslog.d/fixtab.conf

$EscapeControlCharactersOnReceive off

And finally, don’t forget to restart the syslog daemon after you make any configuration changes in order for them to take effect:

service rsyslog restart

Server Log Confguration

Unlike application logs and error logs that you can write to, server logs are exclusively written to by the corresponding server daemons (e.g., web server, database server, etc.) on each request. The only “control” you have over these logs is to the extent that the server allows you to configure its logging functionality. Though there can be a lot to sift through in these files, they are often the only way to get a clear sense of what’s going on “under the hood” with your server.

Let’s deploy our Symfony Standard example application on nginx environment with MySQL storage backend. Here’s the nginx host config we will be using:

server {
    server_name my.log-sandbox;
    root /var/www/my/web;

    location / {
        # try to serve file directly, fallback to app.php
        try_files $uri /app.php$is_args$args;
    }
    # DEV
    # This rule should only be placed on your development environment
    # In production, don't include this and don't deploy app_dev.php or config.php
    location ~ ^/(app_dev|config)\.php(/|$) {
        fastcgi_pass unix:/var/run/php5-fpm.sock;
        fastcgi_split_path_info ^(.+\.php)(/.*)$;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param HTTPS off;
    }
    # PROD
    location ~ ^/app\.php(/|$) {
        fastcgi_pass unix:/var/run/php5-fpm.sock;
        fastcgi_split_path_info ^(.+\.php)(/.*)$;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param HTTPS off;
        # Prevents URIs that include the front controller. This will 404:
        # http://domain.tld/app.php/some-path
        # Remove the internal directive to allow URIs like this
        internal;
    }

    error_log /var/log/nginx/my_error.log;
    access_log /var/log/nginx/my_access.log;
}

With regard to the last two directives above: access_log represents the general requests log, while error_log is for errors, and, as with application error logs, it’s worth setting up extra monitoring to be alerted to problems so you can react quickly.

Note: This is an intentionally oversimplified nginx config file that is provided for example purposes only. It pays almost no attention to security and performance and shouldn’t be used as-is in any “real” environment.

This is what we get in /var/log/nginx/my_access.log after typing http://my.log-sandbox/app_dev.php/ in browser and hitting Enter.

192.168.56.1 - - [26/Apr/2015:16:13:28 +0300] "GET /app_dev.php/ HTTP/1.1" 200 6715 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
192.168.56.1 - - [26/Apr/2015:16:13:28 +0300] "GET /bundles/framework/css/body.css HTTP/1.1" 200 6657 "http://my.log-sandbox/app_dev.php/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
192.168.56.1 - - [26/Apr/2015:16:13:28 +0300] "GET /bundles/framework/css/structure.css HTTP/1.1" 200 1191 "http://my.log-sandbox/app_dev.php/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
192.168.56.1 - - [26/Apr/2015:16:13:28 +0300] "GET /bundles/acmedemo/css/demo.css HTTP/1.1" 200 2204 "http://my.log-sandbox/app_dev.php/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
192.168.56.1 - - [26/Apr/2015:16:13:28 +0300] "GET /bundles/acmedemo/images/welcome-quick-tour.gif HTTP/1.1" 200 4770 "http://my.log-sandbox/app_dev.php/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
192.168.56.1 - - [26/Apr/2015:16:13:28 +0300] "GET /bundles/acmedemo/images/welcome-demo.gif HTTP/1.1" 200 4053 "http://my.log-sandbox/app_dev.php/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
192.168.56.1 - - [26/Apr/2015:16:13:28 +0300] "GET /bundles/acmedemo/images/welcome-configure.gif HTTP/1.1" 200 3530 "http://my.log-sandbox/app_dev.php/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
192.168.56.1 - - [26/Apr/2015:16:13:28 +0300] "GET /favicon.ico HTTP/1.1" 200 6518 "http://my.log-sandbox/app_dev.php/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
192.168.56.1 - - [26/Apr/2015:16:13:30 +0300] "GET /app_dev.php/_wdt/e50d73 HTTP/1.1" 200 13265 "http://my.log-sandbox/app_dev.php/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"

This shows that, for serving one page, the browser actually performs 9 HTTP calls. 7 of those, however, are requests to static content, which are plain and lightweight. However, they still take network resources and this is what can be optimized by using various sprites and minification techniques.

While those optimisations are to be discussed in another article, what’s relavant here is that we can log requests to static contents separately by using another location directive for them:

location ~ \.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|pdf|txt|tar|wav|bmp|rtf|js)$ {
    access_log /var/log/nginx/my_access-static.log;
}

Remember that nginx location performs simple regular expression matching, so you can include as many static contents extensions as you expect to dispatch on your site.

Parsing such logs is no different than parsing application logs.

Other Logs Worth Mentioning

Two other PHP logs worth mentioning are the debug log and data storage log.

The Debug Log

Another convenient thing about nginx logs is the debug log. We can turn it on by replacing the error_log line of the config with the following (requires that the nginx debug module be installed):

error_log /var/log/nginx/my_error.log debug;

The same setting applies for Apache or whatever other webserver you use.

And incidentally, debug logs are not related to error logs, even though they are configured in the error_logdirective.

Although the debug log can indeed be verbose (a single nginx request, for example, generated 127KB of log data!), it can still be very useful. Wading through a log file may be cumbersome and tedious, but it can often quickly provide clues and information that greatly help accelerate the debugging process.

In particular, the debug log can really help with debugging nginx configurations, especially the most complicated parts, like location matching and rewrite chains.

Of course, debug logs should never be enabled in a production environment. The amount of space they use also and the amount of information that they store means a lot of I/O load on your server, which can degrade the whole system’s performance significantly.

Data Storage Logs

Another type of server log (useful for debugging) is data storage logs. In MySQL, you can turn them on by adding these lines:

[mysqld]
general_log = 1
general_log_file = /var/log/mysql/query.log

These logs simply contain a list of queries run by the system while serving database requests in chronological order, which can be helpful for various debugging and tracing needs. However, they should not stay enabled on production systems, since they will generate extra unnecessary I/O load, which affects performance.

Writing to Your Log Files

PHP itself provides functions for opening, writing to, and closing log files (openlog(), syslog(), and closelog(), respectively).

There are also numerous logging libraries for the PHP developer, such as Monolog (popular among Symfonyand Laravel users), as well as various framework-specific implementations, such as the logging capabilities incorporated into CakePHP. Generally, libraries like Monolog not only wrap syslog() calls, but also allow using other backend functionality and tools.

Here’s a simple example of how to write to the log:

<?php

openlog(uniqid(), LOG_ODELAY, LOG_LOCAL0);
syslog(LOG_INFO, 'It works!');

Our call here to openlog:

  • configures PHP to prepend a unique identifier to each system log message within the script’s lifetime
  • sets it to delay opening the syslog connection until the first syslog() call has occurred
  • sets LOG_LOCAL0 as the default logging facility

Here’s what the contents of the log file would look like after running the above code:

# cat /var/log/my/info.log
Mar  2 00:23:29 log-sandbox 54f39161a2e55: It works!

Maximizing the Value of Your PHP Log Files

Now that we’re all good with theory and basics, let’s see how much we can get from logs making as few changes as possible to our sample Symfony Standard project.

First, let’s create the scripts src/log-begin.php (to properly open and configure our logs) and src/log-end.php(to log information about successful completion). Note that, for simplicity, we’ll just write all messages to the info log.

# src/log-begin.php

<?php

define('START_TIME', microtime(true));
openlog(uniqid(), LOG_ODELAY, LOG_LOCAL0);
syslog(LOG_INFO, 'BEGIN');
syslog(LOG_INFO, "URI\t{$_SERVER['REQUEST_URI']}");
$browserHash = substr(md5($_SERVER['HTTP_USER_AGENT']), 0, 7);
syslog(LOG_INFO, "CLIENT\t{$_SERVER['REMOTE_ADDR']}\t{$browserHash}"); <br />

# src/log-end.php

<?php

syslog(LOG_INFO, "DISPATCH TIME\t" . round(microtime(true) - START_TIME, 2));
syslog(LOG_INFO, 'END');

And let’s require these scripts in app.php:

<?php

require_once(dirname(__DIR__) . '/src/log-begin.php');
syslog(LOG_INFO, "MODE\tPROD");

# original app.php contents

require_once(dirname(__DIR__) . '/src/log-end.php');

For the development environment, we want to require these scripts in app_dev.php as well. The code to do so would be the same as above, except we would set the MODE to DEV rather than PROD.

We also want to track what controllers are being invoked, so let’s add one more line in Acme\DemoBundle\EventListener\ControllerListener, right at the beginning of the ControllerListener::onKernelController() method:

syslog(LOG_INFO, "CONTROLLER\t" . get_class($event->getController()[0]));

Note that these changes total a mere 15 extra lines of code, but can collectively yield a wealth of information.

Analyzing the Data in Your Log Files

For starters, let’s see how many HTTP requests are required to serve each page load.

Here’s the info in the logs for one request, based on the way we’ve configured our logging:

Mar  3 12:04:20 log-sandbox 54f58724b1ccc: BEGIN
Mar  3 12:04:20 log-sandbox 54f58724b1ccc: URI    /app_dev.php/
Mar  3 12:04:20 log-sandbox 54f58724b1ccc: CLIENT 192.168.56.1    1b101cd
Mar  3 12:04:20 log-sandbox 54f58724b1ccc: MODE   DEV
Mar  3 12:04:23 log-sandbox 54f58724b1ccc: CONTROLLER Acme\DemoBundle\Controller\WelcomeController
Mar  3 12:04:25 log-sandbox 54f58724b1ccc: DISPATCH TIME  4.51
Mar  3 12:04:25 log-sandbox 54f58724b1ccc: END
Mar  3 12:04:25 log-sandbox 54f5872967dea: BEGIN
Mar  3 12:04:25 log-sandbox 54f5872967dea: URI    /app_dev.php/_wdt/59b8b6
Mar  3 12:04:25 log-sandbox 54f5872967dea: CLIENT 192.168.56.1    1b101cd
Mar  3 12:04:25 log-sandbox 54f5872967dea: MODE   DEV
Mar  3 12:04:28 log-sandbox 54f5872967dea: CONTROLLER Symfony\Bundle\WebProfilerBundle\Controller\ProfilerController
Mar  3 12:04:29 log-sandbox 54f5872967dea: DISPATCH TIME  4.17
Mar  3 12:04:29 log-sandbox 54f5872967dea: END

So now we know that each page load is actually served with two HTTP requests.

Actually there are two points worth mentioning here. First, the two requests per page load is for using Symfony in dev mode (which I have done throughout this article). You can identify dev mode calls by searching for /app-dev.php/ URL chunks. Second, let’s say each page load is served with two subsequent requests to the Symfony app. As we saw earlier in the nginx access logs, there are actually more HTTP calls, some of which are for static content.

OK, now let’s surf a bit on the demo site (to build up the data in the log files) and let’s see what else we can learn from these logs.

How many requests were served in total since the beginning of the logfile?

# grep -c BEGIN info.log
10

Did any of them fail (did the script shut down without reaching the end)?

# grep -c END info.log
10

We see that the number of BEGIN and END records match, so this tells us that all of the calls were successful. (If the PHP script had not completed successfully, it would not have reached execution of the src/log-end.phpscript.)

What was the percentage of requests to the site root?

# `grep -cE "\s/app_dev.php/$" info.log`
2

This tells us that there were 2 page loads of the site root. Since we previously learned that (a) there are 2 requests to the app per page load and (b) there were a total of 10 HTTP requests, the percentage of requests to the site root was 40% (i.e., 2×2/10).

Which controller class is responsible for serving requests to site root?

# grep -E "\s/$|\s/app_dev.php/$" info.log | head -n1
Mar  3 12:04:20 log-sandbox 54f58724b1ccc: URI  /app_dev.php/

# grep 54f58724b1ccc info.log | grep CONTROLLER
Mar  3 12:04:23 log-sandbox 54f58724b1ccc: CONTROLLER   Acme\DemoBundle\Controller\WelcomeController

Here we used the unique ID of a request to check all log messages related to that single request. We thereby were able to determine that the controller class responsible for serving requests to site root is Acme\DemoBundle\Controller\WelcomeController.

Which clients with IPs of subnet 192.168.0.0/16 have accessed the site?

# grep CLIENT info.log | cut -d":" -f4 | cut -f2 | sort | uniq
192.168.56.1

As expected in this simple test case, only my host computer has accessed the site. This is of course a very simplistic example, but the capability that it demonstrates (of being able to analyse the sources of the traffic to your site) is obviously quite powerful and important.

How much of the traffic to my site has been from FireFox?

Having 1b101cd as the hash of my Firefox User-Agent, I can answer this question as follows:

# grep -c 1b101cd info.log
8
# grep -c CLIENT info.log
10

Answer: 80% (i.e., 8/10)

What is the percentage of requests that yielded a “slow” response?

For purposes of this example, we’ll define “slow” as taking more than 5 seconds to provide a response. Accordingly:

# grep "DISPATCH TIME" info.log | grep -cE "\s[0-9]{2,}\.|\s[5-9]\."
2

Answer: 20% (i.e., 2/10)

Did anyone ever supply GET parameters?

# grep URI info.log | grep \?

No, Symfony standard uses only URL slugs, so this also tells us here that no one has attempted to hack the site.

These are just a handful of relatively rudimentary examples of the ways in which logs files can be creatively leveraged to yield valuable usage information and even basic analytics.

Other Things to Keep in Mind

Keeping Things Secure

Another heads-up is for security. You might think that logging requests is a good idea, in most cases it indeed is. However, it’s important to be extremely careful about removing any potentially sensitive user information before storing it in the log.

Fighting Log File Bloat

Since log files are text files to which you always append information, they are constantly growing. Since this is a well-known issue, there are some fairly standard approaches to controlling log file growth.

The easiest is to rotate the logs. Rotating logs means:

  • Periodically replacing the log with a new empty file for further writing
  • Storing the old file for history
  • Removing files that have “aged” sufficiently to free up disk space
  • Making sure the application can write to the logs uniterrupted when these file changes occur

The most common solution for this is logrotate, which ships pre-installed with most *nix distributions. Let’s see a simple configuration file for rotating our logs:

/var/log/my/debug.log
/var/log/my/info.log
/var/log/my/warning.log
/var/log/my/error.log
{
    rotate 7
    daily
    missingok
    notifempty
    delaycompress
    compress
    sharedscripts
    postrotate
        invoke-rc.d rsyslog rotate > /dev/null
    endscript
}

Another, more advanced approach is to make rsyslogd itself write messages into files, dynamically created based on current date and time. This would still require a custom solution for removal of older files, but lets devops manage timeframes for each log file precisely. For our example:

$template DynaLocal0Err,        "/var/log/my/error-%$NOW%-%$HOUR%.log"
$template DynaLocal0Info,       "/var/log/my/info-%$NOW%-%$HOUR%.log"
$template DynaLocal0Warning,    "/var/log/my/warning-%$NOW%-%$HOUR%.log"
$template DynaLocal0Debug,      "/var/log/my/debug-%$NOW%-%$HOUR%.log"
local1.err      -?DynaLocal0Err
local1.info     -?DynaLocal0Info
local1.warning  -?DynaLocal0Warning
local1.debug    -?DynaLocal0Debug

This way, rsyslog will create an individual log file each hour, and there won’t be any need for rotating them and restarting the daemon. Here’s how log files older than 5 days can be removed to accomplish this solution:

find /var/log/my/ -mtime +5 -print0 | xargs -0 rm

Remote Logs

As the project grows, parsing information from logs gets more and more resource hungry. This not only means creating extra server load; it also means creating peak load on the CPU and disk drives at the times when you parse logs, which can degrade server response time for users (or in a worst case can even bring the site down).

To solve this, consider setting up a centralized logging server. All you need for this is another box with UDP port 514 (default) open. To make rsyslogd listen to connections, add the following line to its config file:

$UDPServerRun 514

Having this, setting up the client is then as easy as:

*.*   @HOSTNAME:514

(where HOSTNAME is the host name of your remote logging server).

Conclusion

While this article has demonstrated some of the creative ways in which log files can offer way more valuable information than you may have previously imagined, it’s important to emphasize that we’ve only scratched the surface of what’s possible. The extent, scope, and format of what you can log is almost limitless. This means that – if there’s usage or analytics data you want to extract from your logs – you simply need to log it in a way that will make it easy to subsequently parse and analyze. Moreover, that analysis can often be performed with standard Linux command line tools like grep, sed, or awk.

Indeed, PHP log files are a most powerful tool that can be of tremendous benefit.

Resources

Code on GitHub: https://github.com/isanosyan/toptal-blog-logs-post-example


Appendix: Reading and Manipulating Log Files in the Unix Shell

Here is a brief intro to some of the more common *nix command line tools that you’ll want to be familiar with for reading and manipulating your log files.

  • cat is perhaps the most simple one. It prints the whole file to the output stream. For example, the following command will print logfile1 to the console:
    cat logfile1
    
  • > character allows user to redirect output, for example into another file. Opens target stream in write mode (which means wiping target contents). Here’s how we replace contents of tmpfile with contents of logfile1:
    cat logfile1 > tmpfile
    
  • >> redirects output and opens target stream in append mode. Current contents of target file will be preserved, new lines will be added to the bottom. This will append logfile1 contents to tmpfile:
    cat logfile1 >> tmpfile
    
  • grep filters file by some pattern and prints only matching lines. Command below will only print lines of logfile1 containing Bingo message:
    grep Bingo logfile1
    
  • cut prints contents of a single column (by number starting from 1). By default searches for tab characters as delimiters between column. For example, if you have file full of timestamps in format YYYY-MM-DD HH:MM:SS, this will allow you to print only years:
    cut -d"-" -f1 logfile1
    
  • head displays only the first lines of a file
  • tail displays only the last lines of a file
  • sort sorts lines in the output
  • uniq filters out duplicate lines
  • wc counts words (or lines when used with the -l flag)
  • | (i.e., the “pipe” symbol) supplies output from one command as input to the next. Pipe is very convenient for combining commands. For example, here’s how we can find months of 2014 that occur within a set of timestamps:
    grep -E "^2014" logfile1 | cut -d"-" -f2 | sort | uniq
    

Here we first match lines against regular expression “starts with 2014”, then cut months. Finally, we use combination of sort and uniq to print occurrences only once.

This article originally appeared on Toptal

How to Use E-mail PGP Encryption

Every day internet users send more than 200 billion emails and this statistics makes anonymous e-mail a number one feature to use in day-to-day communication.

Below here is a step by step guide to set up PGP encryption and communicate securely online. The tutorial is developed using Privatoria E-mail service, though one can use any e-mail client supporting this feature. The device used in this installation wizard is MacBook Air, OS X Yosemite. Pay attention that you can set up PGP on any device, regardless of OS.

Step 1

Please login to your Control Panel and go to Services /Anonymous E-Mail / Go to your inbox. You will be redirected to your mail inbox.

Step 2

Go to Settings section in the right top corner of your page. Click Open PGP – Enable OpenPGP – Save.

To start sending encrypted emails you need to first of all generate a KEY and send it to the recipient.

Click on Generate new key. It is recommended to set own password on this step and use it each and every time you want to send/receive encrypted emails. It provides additional security meaning that your PGP email can never be decrypted and/or your  signature is really yours! Please note that this password cannot be reset, but you can create a new one in case you forget it 🙂

Click Generate and wait for about half a minute until it is being generated. Once ready, you will see the following screen

To send an encrypted email you will need to use Public Key. Click View and then choose Send.

Step 3

You will be redirected to email interface ready to send your email. Send this email to the recipient first and wait until they send you their key. Once received click Import. Now, once the key is received you can proceed with sending the actual both-end encrypted email. Type in the recipient’s email, add subject  and body of your email.

To finalize the encryption click on PGP Sign/Encrypt and proceed further. You will receive a message “OpenPGP supports plain text only. Click OK to remove all the formatting and continue.” Click OK.

NB! Private key is your own key that encrypts the data and is being generated for you only.

Email recipient should use any email client supporting PGP encryption function.

 

Cold War Tech: It’s Still Here, And Still Being Used

I’m a Cold War kid. I grew up watching news of Pershing II and SS-20 deployments in Europe, the Soviet war in Afghanistan, with some Terminator and Top Gun VHS action on the side. Yugoslavia was trying to play both sides, and for a while it worked like a charm. It all crashed a couple of years after the Berlin Wall came tumbling down, rendering our unaligned prowess pointless.

I admit this is an odd intro for a tech blog, but bear with me; it will start to make sense. Unlike most Europeans, we had good relations with both blocs. We sold tanks to Kuwait and rocket artillery to Saddam, we bought cheap fuel and MiGs from the Soviets, and in return we exported some stuff they couldn’t get directly from the West. I know people who would stay in East Berlin hotels because they were cheaper, then cross the border into West Berlin to work, play and shop, only to cross back via virtually unused border crossings like Checkpoint Charlie, all in a matter of hours.

On one such trip, my dad got me a Commodore C64, which was pressed into service as our Cold War gaming machine. Most 80s video games, and indeed a lot of music and films, were inspired by countless proxy wars and the threat of a nuclear apocalypse. As the Wall came down, a lot of people assumed that would be the end of runaway defence spending and that the world would be a safer place. It didn’t exactly work out that way, did it?

However, the long-term effect of the Cold War on science and technology is more profound than Nena’s 99 Luftbalons, or any Oliver Stone Vietnam flick.

Minuteman: A Cold War Tech Case Study

If you are reading this, you’re already using a technology developed for cold warriors; The Internet. That’s not all. A lot of tech and infrastructure we take for granted was developed, or at least conceived, during these tumultuous decades.

That constellation of GPS satellites orbiting Earth? It wasn’t put up there to geotag selfies or get an Uber ride; it was designed to help the US Strategic Air Command deliver hundreds of megatons worth of instant sunshine on Soviet targets with pinpoint accuracy. Integrated circuits, transistors, solid-state computing? Yep, all developed for the armed forces and paid for by the US taxpayer.

Here is just one example: the sleek and unfathomably deadly LGM-30 Minuteman intercontinental ballistic missile (ICBM). It wasn’t the first ICBM out there, but when it appeared, it was revolutionary. It was a solid fuel missile, which meant it could respond to a threat and launch in a minute without having to be fuelled, hence the name. But solid fuel was only part of the story: Solid-state was a lot more interesting from a geek perspective. Prior to Minuteman, ICBMs relied on analogue computers with mechanical gyros and primitive sensors. Since they were wired to a specific target, the target package could not be changed easily. Minuteman was the first mass implementation of a general purpose digital computer; it integrated an autopilot and missile guidance system in one package, with reliable storage that could take the stress of a silo launch. The computer was also capable of storing multiple targets, and was reprogrammable.

Transistors were nothing new at that point; they were developed years before by Bell Labs. Yes, these primitive transistors were almost exclusively reserved for the military-industrial complex. Uncle Sam was the sole customer for virtually all early computers and chips, burning heaps of money. These early transistors offered a quantum leap over vacuum tubes, but they weren’t perfect. By today’s standards, they were rubbish. The reliability simply wasn’t there, and if you needed to launch a few hundred thermonuclear warheads halfway across the planet, you sort of needed a guidance system that wouldn’t fail as soon as the candle was lit.

So what do you do when you come across a technical problem you can’t solve with money? Simple: You throw more money at it, and that’s exactly what the US Air Force did. They burned millions to make the damn things reliable enough to be used in harsh environments and survive the stress of a high-G ascent to space. This was known as the Minuteman High Reliability (Hi-Rel) programme.

The first truly mobile digital computer was somewhat deadlier than your notebook and iPhone.

The first truly mobile digital computer was somewhat deadlier than your notebook and iPhone.

It worked, but the USAF got a bit more than they bargained for. In trying to improve a single weapons system, the USAF ended up giving a huge boost to the tech industry in general. Eventually, the Minuteman was upgraded to include a new microchip-based guidance system, with a primitive form of solid-state storage. This Cold War relic has been in service since the Kennedy administration, and the current incarnation has been around for 45 years, receiving multiple hardware and software updates over the years.

So, in outlining the development and evolution of a single strategic weapon delivery system, I have touched on a number of vital technologies we take for granted: reliable transistors, chips, solid-state storage, mass-produced programmable computers and so on. The Minuteman was also the first mobile digital computer.

Some may argue that the legacy of such weapons is that Mutually Assured Destruction (MAD), guaranteed by the nuclear triad, kept superpowers from going to all-out war. It probably did, but in doing so, it also allowed engineers around the world to develop technologies and concepts applicable in various industries and fields of study.

Their real legacy lies in every integrated circuit on the planet.

Capitalist Pioneers Try To Cash In

What could be more capitalist than monetizing instruments of mass murder? The taxpayers paid for their development, not venture capitalists!

Joking aside, it can be argued that the Red Scare of the fifties created Silicon Valley. Most of the money really did come from taxpayers, and most companies that got lucrative defence contracts were quick to make a buck on dual-use technology developed for the military. Remember Bell Labs? A few of their brightest people went on to co-found Fairchild Semiconductor, and eventually created Intel a decade later. The updated Minuteman guidance computer was based on chips from another semiconductor giant: Texas Instruments.

I am not disputing the brilliance of people like Intel co-founders Robert Noyce and Gordon Moore. I have no doubt they would have made their mark on the tech industry even without the biggest arms race in history, but it’s also hard to dispute that the tech industry wouldn’t have developed at nearly the same pace had there been no government funding. Yes, the taxpayers effectively subsidised the tech industry for decades, but in the long run, they’re probably better off. Westinghouse did not need subsidies to develop washing machines and refrigerators, because consumer demand was strong, but in the early days of computing, there was virtually no consumer demand. That’s why governments had to step in.

But what did the taxpayer get?

The Internet, GPS, reliable transistors and chips: Cold War tech made possible by runaway defence spending.

The Internet, GPS, reliable transistors and chips: Cold War tech made possible by runaway defence spending.

The space and arms race spawned a number of technologies that in turn created countless business opportunities. Even primitive computers had a profound impact on industry. They made energy grids and transportation infrastructure more efficient, helped improve safety in industrial facilities, including sensitive chemical and nuclear facilities, they changed the face of banking, communications, entertainment and so on.

Best of all, we somehow managed not to blow ourselves up with the weapons these technologies made possible, yet at the same time, we turned swords into ploughshares. Back in the fifties, the US and USSR launched initiatives designed to examine civilian uses of nuclear power (including civil engineering nuclear explosives schemes, which went terribly wrong), but they amounted to nothing. It wasn’t the might of the atom that changed the world, it was the humble microchip and ancillary technologies developed for countless defence programmes.

Before they made their mark in science and beat Gary Kasparov at the chess table, supercomputers and their analogue predecessors were used to simulate physical processes vital in the development of thermonuclear weapons. An advantage in sheer computing power could yield advances in countless fields. Computer simulations allowed western navies to develop quieter submarines with new screws, digitally optimised to avoid cavitation. Digital signal processors (DSPs) made sonars far more sensitive, and a couple of decades later, advanced DSPs made music sound better. Computer aided design wasn’t just used to reduce the radar cross-section of airplanes, it also made our buildings and cars cheaper, safer and more energy efficient.

Some of these efforts resulted in a technological dead-end, but most did not. One of my favourite tech duds was Blue Peacock, a British nuclear landmine (yes, landmine, not bomb), weighing in at 7.2 tons. Since it relied on early 50s technology and had to be buried in the German countryside, the engineers quickly realised the cold could kill the electronics inside, so they tried to figure out how to keep circuits warm. Their solution was so outlandish that it was mistaken for an April Fool’s Day joke when the design was declassified on April 1, 2004.

No chickens were harmed in the making of this blog post, or in the Blue Peacock nuclear land mine programme.

No chickens were harmed in the making of this blog post, or in the Blue Peacock nuclear landmine programme.

A chicken was to be sealed inside the casing, with enough food and water to stay alive for a week. Its body heat would keep the bomb’s electronics operational.

As civilian industries started implementing these cutting edge technologies en masse, our quality of life and productivity shot up exponentially. Our TVs, cars, phones, the clothes we wear, and just about any consumer product we buy: They’re all better thanks to the biggest waste of money in history. Granted, we all have trace amounts of Strontium 90 in our bones, but in the big scheme of things, it’s a small price to pay for the high-tech world we enjoy so much.

Oh yes, we also got video games out of it. Loads and loads of video games.

Kickstarting Game Development

Video games were pioneered on the earliest digital computers (and some analogue ones as well). In fact,Tennis for Two, arguably the first game to use a graphical display, was developed for an analogue computer in 1958. However, not even Bond villains had computers at that point, so the rise of the video game industry had to wait for hardware to mature.

By the mid to late seventies, microchips became cheap enough for mass market applications. Now that we had the hardware, we just needed some software developers and a use-case for cheap chips. Since the average consumer was not interested in expensive and complicated computers that were designed for big business, attention shifted to gaming; arcades, game consoles and inexpensive computers like the ZX and C64.

These humble machines brought programmable computers to millions of households, hooking a generation of kids on digital entertainment, and creating opportunities for game developers. Consoles and cheap computers brought the arcade to the living room, ushering in a new era of video gaming, and creating countless jobs in the industry. Even the Soviets got in on it with Tetris, the first game from behind the iron curtain.

The advent of inexpensive home computers and game consoles created a generation hooked on computing and coding.

The advent of inexpensive home computers and game consoles created a generation hooked on computing and coding.

It wasn’t just entertainment. Unlike consoles, the ZX and C64 were proper computers, and geeky kids quickly found new uses for them. They started making demos, they started coding. Chances are you know a lot of these kids, and if you’re reading this, you probably work with some of them.

If you’re interested in the development of early video games, and what the Cold War had to do with them, I suggest you check out Nuclear Fruit; a new documentary that’s a must see for all geeks and gamers born in the 70s and early 80s.

These guys and gals went on to develop a new breed of video games, build successful online businesses, create new technologies and revolutionise the digital world, all in the space of a decade. A generation that grew up with the constant threat of nuclear war, enjoying dystopian science fiction, helped make the world a better place. They didn’t develop Skynet, they developed millions of mobile and web apps instead.

So, no Terminators. At least, not yet.

Cold War 2.0 And The Emergence Of New Threats

This is not a geopolitical blog, but if you happen to follow the news, you probably know the world is a messed up place. No, the end of the Cold War didn’t bring an era of peace and stability, and there’s already talk of a “Second Cold War,” or worse, a “hot” war. While most of these worries are nothing more than hype and sensationalism, a number of serious threats remain. The threat of nuclear annihilation is all but gone, but the technology we love so much has created a host of potential threats and issues, ranging from privacy and security, to ethical concerns.

Thankfully, we aren’t likely to see an arms race to rival the one we witnessed in the 20th Century, but we don’t have to. The same technology that makes our lives easier and more productive can also be used against us. The digital infrastructure we rely on for work and play is fragile and can be targeted by criminals, foreign governments, non-state actors, and even lone nutjobs with a grudge.

These new threats include, but are not limited to:

  • Cybercrime
  • State-sponsored cyber warfare
  • Misuse of autonomous vehicle technology
  • Privacy breaches
  • Mass surveillance abuses
  • Use of secure communications for criminal/terrorist activities

All pose a serious challenge and the industry is having trouble keeping up. My argument is simple: We no longer have to develop ground-breaking technology to get an edge in geopolitical struggles, but we will continue to develop technologies and methods of tackling new threats and problems. It’s a vicious circle since these new threats are made possible by our reliance on digital communications and the wide availability of various technologies that can be employed by hostile organisations and individuals.

A new generation of emerging threats is once again rallying industry leaders and governments around a common cause.

A new generation of emerging threats is once again rallying industry leaders and governments around a common cause.

Cybercrime is usually associated with identity theft and credit card fraud, but it’s no longer limited to these fields. The advent of secure communication channels has allowed criminals to expand into new niches. The scene has come a long way since the romanticised exploits of phone phreaks like Steve Wozniak. Some offer hacking for hire, others are willing to host all sorts of illicit content, no questions asked. Some groups specialise in money laundering, darknet drug bazaars, and so on. The biggest threat with this new generation of cybercrime is that you no longer have to possess many skills to get involved. As cybercrime matures, different groups specialise in different activities, and they can be hired.

State-sponsored cyberwarfare poses a serious threat to infrastructure, financial systems, and national security. However, there is really not much an individual can do in the face of these threats, so there’s no point in wasting time on them in this post. Another form of economic warfare could be to deprive a nation or region of Internet access. It has happened before, sometimes by accident, sometimes by government decree and enemy action.

Commercial drones don’t have much in common with their military counterparts. Their range and payload are very limited, and while a military drone can usually loiter over an area for hours on end, the endurance of hobbyist drones is limited to minutes rather than hours. This does not mean they cannot be used for crime; they can still invade someone’s privacy, smuggle drugs across a border, or even carry explosives. Autonomous cars are still in their infancy, so I don’t feel the need to discuss the myriad of questions they will raise.

Privacy remains one of the biggest Internet-related concerns expressed by the average person. This is understandable; we have moved so much of our daily lives to the digital sphere, placing our privacy at risk. People don’t even have to be specifically targeted to have their privacy and personal integrity compromised. Most data that makes its way online is released in the form of massive dumps following a security breach affecting many, if not all, users of a particular online service. People will continue to demand more privacy, and in turn clients will demand more security from software engineers (who aren’t miracle workers and can’t guarantee absolute security and privacy).

Mass surveillance is usually performed by governments and should not represent a threat to the average citizen or business. However, it’s still a potential threat as it can be abused by disgruntled workers, foreign governments, or by way of data breaches. The other problem is the sheer cost to the taxpayer; mass surveillance doesn’t come cheap and we will continue to see more of it.

Most governments wouldn’t bother with mass surveillance and metadata programmes if they weren’t facing very real threats. The same technology developed to keep our communications and online activities private can be abused by all sorts of individuals we wouldn’t like to meet in a dark alley. The list includes multinational crime syndicates, terrorists and insurgents. However, not all of this communication needs to be encrypted and secure. The point of propaganda is to make it widely available to anyone, and the Internet has given every crackpot with a smartphone the biggest megaphone in history, with global reach, free of charge. You can use the Internet to rally a million people around a good cause in a matter of days, but the same principles can be applied to a bad cause. If the target audience is people willing to join a death cult with a penchant for black flags, you don’t need a million people; you just need a few dozen.

The Difference Between Science And Science Fiction

For all their brilliance, the science fiction authors who helped shape popular culture in the 20th Century didn’t see the real future coming. They didn’t exactly envision the Internet, let alone its profound impact on society.

Sorry to burst your bubble, but Terminators and Artificial Intelligence (AI) aren’t a threat yet, and won’t be anytime soon. The real threats are more down to earth, but that does not mean we can afford to ignore them. You don’t need a Terminator to create havoc, all you need is a few lines of really nasty code that can disrupt the infrastructure, causing all sorts of problems. You don’t need a super-intelligent automaton from the future to cause damage. Since eBay doesn’t carry Terminators, it’s a lot easier to use an off-the-shelf drone, programmed to deliver a payload to a specific target: drugs to a trafficker, or an explosive charge to a VIP.

But these aren’t the biggest threats, they’re just potential threats: something for a Hollywood script, not a tech blog.

The real threats are criminal in nature, but they tend to stay in the cyber realm. You don’t have to physically move anything to move dirty money and information online. Law enforcement is already having a hard time keeping up with cybercrime, which seems to be getting worse. While it’s true that the crime rate in developed countries is going down, these statistics don’t paint the full picture. A few weeks ago, the British Office for National Statistics (ONS) reported a twofold increase in the crime rate for England and Wales, totalling more than 11.6 million offences. The traditional crime rate continued to fall, but the statistics included 5.1 million online fraud incidents.

The cost of physical crime is going down, but the cost of cybercrime is starting to catch up. I strongly believe the industry will have to do more to bolster security, and governments will have to invest in online security and crime prevention as well.

Just in case you are into dystopian fiction and don’t find criminal threats exciting, another frightening development would be data monopolisation: A process in which industry giants would command such a competitive lead, made possible by their vast user base, as to render competition pointless, thus stifling innovation.

Yes, I am aware that Terminators would make for a more eventful future and interesting blog post, but we’re not there yet.

Source: Toptal

What To Look Out For In Software Development NDAs

The demand for technical talent, and the ease with which information can be shared, has increased entrepreneurs’ reliance on business relationships with outsiders. It has never been easier for an entrepreneur to find, meet, communicate and eventually enter into some sort of business relationship with an individual or company that is otherwise not associated with the business.

ndas and software development

Moreover, sky-high valuations and fairytale overnight success stories have fueled the notion that even a basic idea can be worth millions, if not billions, in a relatively short time. In light of these factors, you might presume that Non-Disclosure Agreements (NDAs) have been widely accepted in the tech world as a means to protect sensitive and potentially valuable information from theft and abuse. Not so fast.

Before jumping into the debate, though, it helps to have a quick understanding of what an NDA is, what one looks like and, eventually, what to look out for if you’re asked to sign one as a freelance software developer.

What Is A Non-Disclosure Agreement?

An NDA is exactly what its name implies — a legal agreement between two or more parties that (i) defines certain confidential information that will be disclosed and (ii) imposes a legal obligation on the receiving party to keep that information confidential. NDAs are most commonly used when a business relationship between two companies or individuals requires the sharing of confidential information.

For example:

Company A, a local retailer, has hired ABC IT Co. to build an online inventory and order management system. To build the system, Company A must provide ABC IT Co. with a list of Company A’s suppliers and certain pricing information. Before disclosing its supplier list and pricing information, Company A asks ABC IT Co. to sign an NDA forbidding ABC IT Co. from disclosing or using Company A’s confidential information.

If a party to an NDA breaches the agreement, by disclosing or using confidential information for example, the other party to the NDA may sue the breaching party for monetary damages (compensation for lost profits or business), injunctive relief (a court order requiring the breaching party to refrain from taking some action) or specific performance (a court order requiring that the breaching party take some specified action).

So What Do Software Development NDAs Look Like?

NDAs are negotiated legal agreements that can be as simple or as complex as the parties desire. An NDA can be a one page fill-in-the-blank form or a lengthy document drafted from scratch to reflect the unique circumstances of the parties’ relationship, the different negotiating leverage of each party, and the nature of the information that will be disclosed.

what ndas look like

Although there is no such thing as a one-size-fits-all software NDA, for purposes of this overview, and to understand generally how NDAs work, it’s important to appreciate the three “main-event” provisions that are common to all NDAs.

(a) The Definition of Confidential Information:

The definition of “Confidential Information” will set forth the type of disclosed information that is subject to the limitations on use and disclosure and, importantly, the type of disclosed information that is not subject to such limitations.

(b) The Term of the Recipient’s Obligations

The term of an NDA sets forth the time limit on the parties’ obligations. The term of an NDA may be measured in days, weeks, months or years depending on the circumstances of the relationship and the nature of the disclosed information.

(c) The Limitation on Use and Disclosure:

This provision will describe what a recipient party may do and what a recipient party may not do with disclosed information that falls within the definition of Confidential Information. This provision will almost certainly forbid disclosure of Confidential Information, but may also limit the use of Confidential Information and, in some cases, require that the recipient take certain affirmative steps to protect the confidentiality of Confidential Information.

The NDA Debate: Should You Ask For An NDA? Should You Sign One?

Although NDAs have been around for as long as there has been information worth protecting, the high-tech startup boom has thrust their use into the limelight and sparked a debate as to their value. As an industry that is highly dependent on data and constantly evolving technology, one would think that the high-tech startup world would embrace the use of software development NDAs. To understand why that isn’t the case, and to better gauge whether you should ask for an NDA or sign one presented to you, consider the following:

NDAs Are Often Unilateral

NDAs are unilateral when the business relationship requires that only one party disclose confidential information (rather than a mutual exchange of information by each party).

A startup seeks to hire an engineer to build its mobile app and has asked the engineer to sign an NDA. The startup will disclose information to the engineer, but the relationship does not require the engineer to provide confidential information to the startup. The NDA will be unilateral and will impose legal obligations, and potential liability, on the engineer only.

Because only one party is exchanging confidential information, only one party (the recipient party) has a legal obligation to comply with and, as such, only the recipient party is subject to potential liability. What an entrepreneur might view as a means by which to protect an idea, an NDA recipient might view as a one-sided contract.

Entrepreneurs Often Overstate the Need for an NDA

There are surely circumstances where NDAs make sense. Customer lists, pricing information, proprietary formulas and algorithms might have intrinsic value that is best protected by an NDA. Many argue, however, that some entrepreneurs are NDA trigger happy and think that every idea is worthy of legal protection. Ideas though, it is argued, are rarely new and, moreover, often have no value without execution.

NDAs should be asked for only when there is something worth protecting, and many argue that an idea alone does not warrant asking for an NDA. Finally, those most often asked to sign software NDAs – investors and engineers – rarely have any interest in stealing an idea when doing so would likely ruin any professional goodwill and reputation they’ve earned in their respective professional communities.

NDAs Indicate Mistrust

In a perfect world, business would be business and would never be personal. In reality, though, business is often about perception. What might be “just a contract” to an entrepreneur asking for a software development NDA, may be perceived as an indication of mistrust and a questioning of personal integrity by the person being asked to sign one. NDAs are most often requested at the outset of a business relationship, signaling mistrust and calling into question one’s professional integrity may start the relationship off on the wrong foot—even if that wasn’t the intention…perception is powerful.

This issue is less of a concern for business relationships where both parties will be disclosing confidential information and, thus, an NDA will be bilateral and both parties subject to legal obligation. Outside of strategic joint ventures, partnerships, mergers and similar arrangements, however, bilateral exchanges are rare and unilateral NDAs are much more common.

NDAs Can Limit An Information Recipient’s Ability To Earn A Living

As discussed earlier, an NDA defines a set of information that is to be considered “Confidential Information” and then specifies what a recipient may and may not do with that information during the term of the NDA. Whether an NDA is three pages or three-hundred pages, no contract can predict and plan for every possible circumstance and this limitation often works against the recipient of disclosed information.

What if, after signing an NDA, an engineer is asked to build a similar product or to execute a similar but technically different idea? Is using similar code on a different application a violation of the NDA’s non-use provision? What if the engineer learned new skills during the engagement? Can the engineer use those skills for another client? Can the engineer list the client on his or her resume?

There is a real concern that signing even one NDA, whether as an engineer, an investor or otherwise, can drastically shrink one’s pool of potential business. At worst, signing an NDA might foreclose a person’s ability to work on even slightly related projects. At best, signing an NDA complicates future business development efforts as every new opportunity requires a time consuming analysis of conflicts and liability under each and every NDA that the person may be subject to.

Enforcement Isn’t Cheap

The whole point of entering into an NDA is to have some legal remedy if the recipient party discloses confidential information in violation of the agreement. An NDA gives a disclosing party a basis to file a lawsuit seeking money damages and/or a court order against the breaching party. What many NDA proponents don’t fully appreciate, however, is the cost of enforcement.

Filing a lawsuit can be extremely costly and time consuming. A lawsuit for breach of contract will very likely require hiring a lawyer to gather evidence, assess possible legal claims, file the initial complaint and supporting documents, depose the allegedly breaching party and any witnesses and related parties, and argue the case before a judge. Lawsuits can take years, and lawyers typically charge by the hour. Before asking for an NDA, one should assess whether the information to be protected is more valuable than the potential cost of enforcement.

Though the above factors have contributed to a move away from NDAs in the startup world, NDAs are not without their value. Whether you should ask for an NDA before disclosing information, or agree to sign one if you’re on the receiving end of the equation, depends on the particular circumstances of the intended business relationship and each party’s motivation to enter into the relationship. The more valuable the relationship is to a party, the less leverage that party has to negotiate for or against the use of an NDA. The less valuable the relationship is to a party, the more leverage that party has to get its way or walk away. This push and pull is at the heart of all negotiations, the party with the better “Best Alternative to a Negotiated Agreement (BATNA)” has the upper hand.

If You Must Have An NDA…

So what if you’re an engineer and the opportunity to work on a particular project outweighs the risk of signing a software development NDA? What if you’re a startup and the intrinsic value of your information justifies the need for an NDA, despite the difficulty of finding an engineer that will sign one? If you have to sign an NDA, or if you must ask for one, what are some of the things to look out for and consider?

nda without reading

As is always the case, I strongly suggest seeking the guidance of a competent and licensed attorney.Contracts can get complex quickly and legal rights and obligations shouldn’t be left to “winging it.” As you’re finding an attorney, though, you can start by reviewing the some of the NDA’s main operative provisions. The following are a few preliminary things you might consider when presented with or requesting an NDA:

1. Definition of Confidential Information

Recall that this provision defines the type of disclosed information that is subject to the confidentiality obligations of the NDA and, as such, it should reflect the nature of the business relationship and that of the information to be disclosed.

If you’re a disclosing party, you’ll likely ask for a broad definition of Confidential Information to cover everything that might be disclosed to the receiving party during the course of the relationship. If you’re a receiving party, however, you might resist this request and seek instead to narrow the definition to include only specifically designated information such as, for example, written information that is marked “Confidential.” Regardless of where the negotiations come out, the parties should think carefully about striking the right balance between a definition of Confidential Information that is too broad (and thus extremely restrictive to the recipient party), on the one hand, and, on the other hand, too narrow (thus minimizing the protective effect to the disclosing party).

Though it’s important to determine the information that is to be held in confidence, it is equally important to “carve-out” certain information that is not subject to the confidentiality provisions. Common examples of such carve-outs include information that is or becomes publicly available and information that is lawfully known before entry into the business relationship.

2. Term of Confidentiality

The term of an NDA should reflect the nature of the parties’ business relationship and the nature of the information to be disclosed. If the relationship is limited to a one-year engagement, it might not make sense for the term of the NDA to extend too far after termination of the relationship. Similarly, certain types of information become less valuable or sensitive over time. Financial statements, for example, may be particularly valuable at and immediately after the time they are prepared, but probably don’t accurately reflect a company’s financial health months or years after their preparation. If information is of a type that decreases in value or sensitivity over time, a long term is likely not necessary.

3. Disclosure to Representatives

As discussed throughout this article, NDAs are typically signed by a single disclosing party and a single recipient party. The problem, though, is that a recipient party may not always work alone and, rather, may from time to time need to disclose information protected by an NDA to such recipient party’s colleagues, employees or representatives in order to carry out the terms of the business relationship.

David Developer has signed an NDA with BigCo to create a mobile app for BigCo.

During the project, David needs to enlist the help of his colleague, Peter Programmer, to write some code in a language that David is less familiar with. Peter has not signed an NDA, can David disclose information to Peter so that Peter can assist with the project?

Rather than go through the hassle of signing a new NDA for each new person to whom information needs to be disclosed during the course of a project, or trying to predict ahead of time every person to whom information may need to be disclosed, the parties to an NDA may include a representatives provision addressing permitted disclosures to certain defined persons.

The representatives provision is straightforward from a drafting perspective and is simply a definition of “Representatives” that specifies the persons or classes of persons to whom confidential information may be disclosed. A recipient party will likely want the definition to be broad and inclusive of any person with whom the recipient party may collaborate. The disclosing party, of course, will likely want to keep the definition of Representatives as narrow as possible to permit the project to move forward, on the one hand, while maintaining the protections of the software development NDA, on the other. Finally, the disclosing party will very likely wish to include a clause providing that, prior to any disclosure of confidential information to a Representative, the recipient party inform such Representative of the confidential nature of the information and of the terms of the NDA. A representatives clause may look something like the following:

During the Term of this Agreement, the Recipient Party will not disclose the Confidential Information to any person other than the Representatives, provided that, prior to any such disclosure to a Representative, the Recipient Party informs such Representative of the confidential nature of the information and the terms of this Agreement. “Representatives” shall include the employees, independent contractors, partners, agents and other third parties that are or may be engaged by the Recipient Party for purposes of the Project.

4. Non-Disclosure v. Non-Use

This is a big one. As mentioned earlier, NDAs will almost always include a prohibition on disclosure of Confidential Information. Some software NDAs, however, will also prohibit or limit use of Confidential Information. For example:

The Recipient Party agrees that, during the Term of this Agreement, the Recipient Party will not (i) disclose the Confidential Information to any person other than its Representatives and (ii) will not use the Confidential Information for any purpose other than for those purposes directly related to the Project.

Depending on the term of the NDA and the type of information disclosed, restriction on use may not be an issue. If the term is particularly long, however, or the definition of Confidential Information particularly broad, the “use prohibition” may be extraordinarily restrictive on the recipient party. For example, consider the following definition of Confidential Information:

“Confidential Information” includes (i) all information furnished by the Disclosing Party to the Recipient Party, whether furnished before or after the date of this Agreement, whether oral or written, and regardless of the manner in which it was furnished, and (ii) all analyses, compilations, forecasts, studies, interpretations, documents, code and similar work product prepared by the Recipient Party or its Representatives in connection with the Project.

What this means is that, for as long as the NDA is in effect, the Recipient Party cannot disclose or use anyinformation that the Disclosing Party made available to the Recipient Party or any information prepared in connection with the particular Project. Without any carve-outs or qualifications, these clauses could be incredibly limiting.

An engineer signs an NDA which includes the two provisions set out above. During the course of the Project, the engineer learns a new way of putting together common strings of code. The new method could be considered work product that was prepared in connection with the Project and, as such, the engineer may be prohibited from using the method in future projects during the term of the NDA.

Before hearing Startup A’s pitch, an investor signs an NDA which includes provisions similar to those set out above. During the pitch, Startup A reveals its most recent financial statements and its strategy for growth. The investor does not invest. A few months later, the investor is approached by a similar startup, Startup B, and asked to attend a pitch. The investor may be precluded from investing in Startup B as doing so might involve use of information learned during Startup A’s pitch, even if only remembered by the investor.

The above examples are admittedly extreme, but are used to stress the point that the combination of a broad definition of Confidential Information, an unnecessarily long term, and restrictions on use can be paralyzing. Additionally, these are by no means the only red-flags that can sneak into an NDA and what might be a red-flag for one NDA may be perfectly tolerable for a different business relationship.

understanding ndas

So What Do I Do…Specifically?

Though you might now have a better understanding of what an NDA is, what a software development NDA might look like, and why many in the tech world are reluctant to sign them, you might still be wondering what, specifically, you should do when on the receiving end of an NDA. There is no substitute for the advice of a competent attorney, but, with an understanding of the concepts discussed in this article, you can approach the first read of an NDA armed with some knowledge as to what is most important to watch for:

  • Is this a bilateral or unilateral NDA? Will both parties be disclosing information? If so, are the parties subject to identical limitations and requirements?
  • How broad, or narrow, is the definition of Confidential Information?
  • How long are the obligations in effect? Does the term of the NDA match the nature of the business relationship and the information to be disclosed?
  • Am I only prohibited from disclosing the Confidential Information, or disclosing and using the Confidential Information?
  • Am I permitted to disclose the information to my employees and colleagues who may assist with the project?
  • Is this relationship valuable enough to assume a legal obligation that can be enforced in a court?

Finally, the above considerations, and this write-up generally, are not solely for the benefit of those who may be asked to sign an NDA. Certainly, a recipient party should consider very carefully an NDA’s provisions before signing, but a party considering asking for an NDA, too, would be wise to consider these factors.

NDAs, like most contracts, have the most value, and are therefore most likely to be signed, when both parties are comfortable with the balance of risks managed by the NDA and the benefit to be realized by the underlying contractual relationship. By considering the perspective of the recipient party, a party asking for an NDA may be better able to tailor the scope of an NDA to match the business relationship and present to the recipient party a fair and balanced agreement.

Though the information in this write-up should give you a good starting point, there is a lot to consider when asking for or presented with an NDA. A competent attorney can work with both parties to draft an NDA that is protective to the disclosing party, without being overly restrictive to the recipient party, and help move the parties towards a mutually beneficial business relationship.

If you want to learn more about legal issues faced by startups and developers, I suggest you check outStartup Law Hacks as well.

Disclaimer: the contents of this article were written and are made available solely as general information and for educational purposes and not to provide specific legal advice of any kind or to establish an attorney-client relationship. This article should not be used as a substitute for competent legal advice from an attorney licensed in your jurisdiction. This article has been written by Bret Stancil in his individual capacity and the views and opinions expressed herein are his own.

Source: Toptal

Integrating Facebook Authentication in AngularJS App with Satellizer

With the arrival of feature-rich front-end frameworks such as AngularJS, more and more logic is being implemented on the front-end, such as data manipulation/validation, authentication, and more. Satellizer, an easy to use token-based authentication module for AngularJS, simplifies the process of implementing authentication mechanism in AngularJS, The library comes with built-in support for Google, Facebook, LinkedIn, Twitter, Instagram, GitHub, Bitbucket, Yahoo, Twitch, and Microsoft (Windows Live) accounts.

Integrating Facebook Login in AngularJS App with Satellizer

In this article, we will build a very simple webapp similar to the one here which allows you to login and see current user’s information.

Authentication vs Authorization

These are 2 scary words that you often encounter once your app starts integrating a user system. According to Wikipedia:

Authentication is the act of confirming the truth of an attribute of a single piece of data (a datum) claimed true by an entity.

Authorization is the function of specifying access rights to resources related to information security and computer security in general and to access control in particular.

In layman terms, let’s take an example of a blog website with some people working on it. The bloggers write articles and the manager validates the content. Each person can authenticate (login) into the system but their rights (authorisation) are different, so the blogger cannot validate content whereas the manager can.

Why Satellizer

You can create your own authentication system in AngularJS by following some tutorials such as this very detailed one: JSON Web Token Tutorial: An Example in Laravel and AngularJS. I suggest reading this article as it explains JWT (JSON Web Token) very well, and shows a simple way to implement authentication in AngularJS using directly the local storage and HTTP interceptors.

So why Satellizer? The principal reason is that it supports a handful of social network logins such as Facebook, Twitter, etc. Nowadays, especially for websites used on mobile, typing username and password is quite cumbersome and users expect to be able to use your website with little hindrance by using social logins. As integrating the SDK of each social network and following their documentations is quite repetitive, it would be nice to support these social logins with minimal effort.

Moreover Satellizer is an active project on Github. Active is key here as these SDKs change quite frequently and you don’t want to read their documentation every now and then (anyone working with Facebook SDK knows how annoying it is)

AngularJS App with Facebook Login

This is where things start to become interesting.

We will build a web app that has regular login/register (i.e. using username, password) mechanism and supports social logins as well. This webapp is very simple as it has only 3 pages:

  • Home page: anyone can see
  • Login page: to enter username/password
  • Secret page: that only logged in users can see

For backend, we will use Python and Flask. Python and the framework Flask are quite expressive so I hope porting the code to other languages/frameworks will not be very hard. We will, of course, use AngularJS for front-end. And for the social logins, we will integrate with Facebook only as it is the most popular social network at this time.

Let’s start!

Step #1: Bootstrap Project

Here is how we will structure our code:

- app.py
- static/
	- index.html
- app.js
	- bower.json
	- partials/
		- login.tpl.html
		- home.tpl.html
		- secret.tpl.html

All the back-end code is in app.py. The front-end code is put in static/ folder. By default, Flask will automatically serve the contents of static/ folder. All the partial views are in static/partials/ and handled by the ui.router module.

To start coding the back-end, we’ll need Python 2.7.* and install the required libraries using pip. You can of course use virtualenv to isolate a Python environment. Below is the list of required Python modules to put in requirements.txt:

Flask==0.10.1
PyJWT==1.4.0
Flask-SQLAlchemy==1.0
requests==2.7.0

To install all these dependencies:

pip install -r requirements.txt

In app.py we have some initial code to bootstrap Flask (import statements are omitted for brevity):

app = Flask(__name__)

@app.route('/')
def index():
    return flask.redirect('/static/index.html')

if __name__ == '__main__':
    app.run(debug=True)

Next we init bower and install AngularJS and ui.router:

bower init # here you will need to answer some question. when in doubt, just hit enter :)
bower install angular angular-ui-router --save # install and save these dependencies into bower.json

Once these libraries are installed, we need to include AngularJS and ui-router in index.html and create routings for 3 pages: home, login, and secret.

<body ng-app="DemoApp">

<a ui-sref="home">Home</a>
<a ui-sref="login">Login</a>
<a ui-sref="secret">Secret</a>
<div ui-view></div>

<script src="bower_components/angular/angular.min.js"></script>
<script src="bower_components/angular-ui-router/release/angular-ui-router.min.js"></script>
<script src="main.js"></script>
</body>

Below is the code that we need in main.js to configure routing:

var app = angular.module('DemoApp', ['ui.router']);

app.config(function ($stateProvider, $urlRouterProvider) {
  $stateProvider
    .state('home', {
      url: '/home',
      templateUrl: 'partials/home.tpl.html'
    })
    .state('secret', {
      url: '/secret',
      templateUrl: 'partials/secret.tpl.html',
    })
    .state('login', {
      url: '/login',
      templateUrl: 'partials/login.tpl.html'
    });
  $urlRouterProvider.otherwise('/home');

});

At this point if you run the server python app.py, you should have this basic interface at http://localhost:5000

The links Home, Login, and Secret should work at this point and show the content of the corresponding templates.

Congratulation, you just finished setting up the skeleton! If you encounter any error, please check out thecode on GitHub

Step #2: Login and Register

At the end of this step, you’ll have a webapp that you can register/login using email and password.

The first step is to configure the backend. We need a User model and a way to generate the JWT token for a given user. The User model shown below is really simplified and does not perform even any basic checks such as if field email contains “@”, or if field password contains at least 6 characters, etc.

class User(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    email = db.Column(db.String(100), nullable=False)
    password = db.Column(db.String(100))

    def token(self):
        payload = {
            'sub': self.id,
            'iat': datetime.utcnow(),
            'exp': datetime.utcnow() + timedelta(days=14)
        }
        token = jwt.encode(payload, app.config['TOKEN_SECRET'])
        return token.decode('unicode_escape')

We use the jwt module in python to generate the payload part in JWT. The iat and exp part correspond to the timestamp that token is created and expired. In this code, the token will be expired in 2 weeks.

After the model User was created, we can add the “login” and “register” endpoints. The code for both are quite similar, so here I will just show the “register” part. Please note that by default, Satellizer will call the endpoints /auth/login and /auth/signup for the “login” and “register” respectively.

@app.route('/auth/signup', methods=['POST'])
def signup():
    data = request.json

    email = data["email"]
    password = data["password"]

    user = User(email=email, password=password)
    db.session.add(user)
    db.session.commit()

    return jsonify(token=user.token())

Let’s check the endpoint using curl first:

curl localhost:5000/auth/signup -H "Content-Type: application/json" -X POST -d '{"email":"[email protected]","password":"xyz"}'

The result should look like this:

{
  "token": "very long string…."
}

Now that the back-end part is ready, let’s attack the front-end! First, we need to install satellizer and add it as a dependency in main.js:

bower install satellizer --save

Add satellizer as dependency:

var app = angular.module('DemoApp', ['ui.router', 'satellizer']);

Login and signup in satellizer is actually quite simple in comparison to all the setup until now:

$scope.signUp = function () {
    $auth
      .signup({email: $scope.email, password: $scope.password})
      .then(function (response) {
        // set the token received from server
        $auth.setToken(response);
        // go to secret page
        $state.go('secret');
      })
      .catch(function (response) {
        console.log("error response", response);
      })
  };

If you have any difficulty setting up the code, you can take a look at the code on GitHub.

Step #3: But Secret View Is Not Really Secret, Because Anyone Can See It!

Yes, that is correct! Until now, anyone can go to secret page without logging in.

It’s time to add some interceptor in AngularJS to make sure that if someone goes to secret page and if this user is not logged in, they will be redirected to the login page.

First, we should add a flag requiredLogin to distinguish secret page from other ones.

    .state('secret', {
      url: '/secret',
      templateUrl: 'partials/secret.tpl.html',
      controller: 'SecretCtrl',
      data: {requiredLogin: true}
    })

The “data” part will be used in the $stateChangeStart event which is fired each time the routing changes:

app.run(function ($rootScope, $state, $auth) {
  $rootScope.$on('$stateChangeStart',
    function (event, toState) {
      var requiredLogin = false;
      // check if this state need login
      if (toState.data && toState.data.requiredLogin)
        requiredLogin = true;
      
      // if yes and if this user is not logged in, redirect him to login page
      if (requiredLogin && !$auth.isAuthenticated()) {
        event.preventDefault();
        $state.go('login');
      }
    });
});

Now, the user cannot go directly to the secret page without logging in. Hooray!

As usual, the code of this step can be found here.

Step #4: It’s Time to Get Something Really Secret!

At this moment, there’s nothing really secret in the secret page. Let’s put something personal there.

This step starts by creating an endpoint in the back-end which is only accessible for an authenticated user, such as having a valid token. The endpoint /user below returns the user_id and email of the user corresponding to the token.

@app.route('/user')
def user_info():
    # the token is put in the Authorization header
    if not request.headers.get('Authorization'):
        return jsonify(error='Authorization header missing'), 401
    
    # this header looks like this: “Authorization: Bearer {token}”
    token = request.headers.get('Authorization').split()[1]
    try:
        payload = jwt.decode(token, app.config['TOKEN_SECRET'])
    except DecodeError:
        return jsonify(error='Invalid token'), 401
    except ExpiredSignature:
        return jsonify(error='Expired token'), 401
    else:
        user_id = payload['sub']
        user = User.query.filter_by(id=user_id).first()
        if user is None:
            return jsonify(error='Should not happen ...'), 500
        return jsonify(id=user.id, email=user.email), 200
    return jsonify(error="never reach here..."), 500

Again, we make use of the module jwt to decode the JWT token included in the ‘Authorization’ header and to handle the case when the token is expired or not valid.

Let’s test this endpoint using curl. First, we need to get a valid token:

curl localhost:5000/auth/signup -H "Content-Type: application/json" -X POST -d '{"email":"[email protected]","password":"xyz"}'

Then with this token:

curl localhost:5000/user -H "Authorization: Bearer {put the token here}"

Which gives this result:

{
  "email": "[email protected]",
  "id": 1
}

Now we need to include this endpoint in the Secret Controller. This is quite simple as we just need to call the endpoint using the regular $http module. The token is automatically inserted to the header by Satellizer, so we don’t need to bother with all the details of saving the token and then putting it in the right header.

  getUserInfo();

  function getUserInfo() {
    $http.get('/user')
      .then(function (response) {
        $scope.user = response.data;
      })
      .catch(function (response) {
        console.log("getUserInfo error", response);
      })
  }

Finally, we have something truly personal in the secret page!

The code of this step is on GitHub.

Step #5: Facebook Login with Satellizer

A nice thing about Satellizer, as mentioned at the beginning, is it makes integrating social login a lot easier. At the end of this step, users can login using their Facebook account!

First thing to do is to create an application on Facebook developers page in order to have an application_idand a secret code. Please follow developers.facebook.com/docs/apps/register to create a Facebook developer account if you don’t have one already and create a website app. After that, you will have the application ID and application secret as in the screenshot below.

Once the user chooses to connect with Facebook, Satellizer will send an authorization code to the endpoint/auth/facebook. With this authorization code, the back-end can retrieve an access token from Facebook/oauth endpoint that allows the call to Facebook Graph API to get user information such as location, user_friends, user email, etc.

We also need to keep track of whether a user account is created with Facebook or through regular signup. To do so, we add facebook_id to our User model.

facebook_id = db.Column(db.String(100)) 

The facebook secret is configured via env variables FACEBOOK_SECRET that we add to app.config.

app.config['FACEBOOK_SECRET'] = os.environ.get('FACEBOOK_SECRET')

So to launch the app.py, you should set this env variable:

FACEBOOK_SECRET={your secret} python app.py

Here is the method which handles Facebook logins. By default Satellizer will call the endpoint /auth/facebook.

@app.route('/auth/facebook', methods=['POST'])
def auth_facebook():
    access_token_url = 'https://graph.facebook.com/v2.3/oauth/access_token'
    graph_api_url = 'https://graph.facebook.com/v2.5/me?fields=id,email'

    params = {
        'client_id': request.json['clientId'],
        'redirect_uri': request.json['redirectUri'],
        'client_secret': app.config['FACEBOOK_SECRET'],
        'code': request.json['code']
    }

    # Exchange authorization code for access token.
    r = requests.get(access_token_url, params=params)
    # use json.loads instead of urlparse.parse_qsl
    access_token = json.loads(r.text)

    # Step 2. Retrieve information about the current user.
    r = requests.get(graph_api_url, params=access_token)
    profile = json.loads(r.text)

    # Step 3. Create a new account or return an existing one.
    user = User.query.filter_by(facebook_id=profile['id']).first()
    if user:
        return jsonify(token=user.token())

    u = User(facebook_id=profile['id'], email=profile['email'])
    db.session.add(u)
    db.session.commit()
    return jsonify(token=u.token())

To send a request to the Facebook server, we use the handy module requests. Now the difficult part on the back-end is done. On the front-end, adding Facebook login is quite simple. First, we need to tell Satellizer ourfacebook_id by adding this code into app.config function:

$authProvider.facebook({
    clientId: {your facebook app id},
    // by default, the redirect URI is http://localhost:5000
    redirectUri: 'http://localhost:5000/static/index.html'
  });

To login using Facebook, we can just call:

$auth.authenticate(“facebook”)

As usual, you can check the code on GitHub

At this time, the webapp is complete in terms of functionality. The user can login/register using regular email and password or by using Facebook. Once logged in, the user can see his secret page.

Make a Pretty Interface

The interface is not very pretty at this point, so let’s add a little bit of Bootstrap for the layout and the angular toaster module to handle an error message nicely, such as when login fails.

The code for this beautifying part can be found here.

Conclusion

This article shows a step-by-step integration of Satellizer in a (simple) AngularJS webapp. With Satellizer, we can easily add other social logins such as Twitter, Linkedin, and more. The code on the front-end is quite the same as in the article. However, the back-end varies as social network SDKs have different endpoints with different protocols. You can take a look at https://github.com/sahat/satellizer/blob/master/examples/server/python/app.py which contains examples for Facebook, Github, Google, Linkedin, Twiter and Bitbucket. When in doubt, you should take a look at the documentation on https://github.com/sahat/satellizer.

This article was written by Son Nguyen Kim, a Toptal freelance developer.