Swift

by

Mocking, Python

I have used Python mock and patch to do some testing for the OPTIONS call and also the server type check in Swift. unittest.mock is a library for testing in Python. It can be used to replace parts of the code, return data and then make assertions about them.

class unittest.mock.Mock(side_effect=None, return_value=DEFAULT, **kwargs)

side_effect: A function to be called whenever the Mock is called. It is useful to raise exceptions.
return_value: The value returned when the mock is called.

Mock provides a patching module that could be used to patch a specific function, class, class level attributes in the scope of test. For instance,

@mock.patch(‘eventlet.green.urllib2.urlopen’)
def test_foo(self, mock_urlopen):
def getheader(name):
d = {‘Server’: ‘server-type’}
return d.get(name)
mock_urlopen.return_value.info.return_value.getheader = getheader
…

In the above function that I used a mock patch to mock urlopen and it’s return value. To test the OPTIONS call in swift-recon, this simulation will faciliate me by returning header value of “Server” as “server-type”. Further, I could assert the return of the actual value returned by the scout_server_type method in recon (which calls OPTIONS), like this:

self.assertEqual(content, ‘server-type’)

Content here is the return value of OPTIONS call in recon.py.

To test the validate servers method (server_type_check) call, I could have created a fake class for scout_server_type or mock the method itself. I have done the latter:

def mock_scout_server_type(app, host):
url = ‘http://%s:%s/’ % (host[0], host[1])
response = responses[host[1]]
status = 200
return url, response, status

patches = [
mock.patch(‘swift.cli.recon.Scout.scout_server_type’,
mock_scout_server_type),
mock.patch(‘sys.stdout’, new=stdout),
]

This mock, will allow me to replace the scout_server_type call and instead call “mock_scout_server_type” and returns those method’s values for the scope of the test.

with nested(*patches):
self.recon.server_type_check(hosts)

With the above, when server_type_check is called, the internal call to scout_server_type will be replaced by mock_scout_server_type.

Advertisement
by

Swift-recon

In continuation with the last post – the recent development is I have build some tests to test the base class. Also added a Server header for OPTIONS. Server is a standard header to inform the client about the type of software used by origin server.[1]

The goal is to add a feature to swift-recon cli to validate servers in the ring. Swift-recon cli is a command line utility to obtain various metrics and telemetry from the servers. To validate servers, I need to get the Server type by calling OPTIONS. This support is added in swift-recon to call OPTIONS (only GET is supported until now). The new command to validate ring will be

–validate-servers

The server_type_check method which I newly created will be called when the command is given. This fetches all the hosts and loops through them by calling scout_server_type (in Scout) which in turn calls OPTIONS on the host:

for url, response in self.pool.imap(recon.scout_server_type, hosts)

imap is a itertool [2] which creates iterator for looping. In the above line, “recon.scout_server_type” is called for each of the “hosts”.

The method responds to the arguments – object/container/account and verify if the servers in the ring indeed match to the argument given. For instance,

swift-recon object –validate-servers

For the above command, it will verify if all of the servers are of object type (by passing the host name to OPTIONS and getting the Server header). If it is not, it displays what they are – container/account.

This needs tests to verify. This post will be continued in the next by detailing the tests and/or any obstacles.

References:
[1] http://tools.ietf.org/html/rfc7231#section-7.4.2
[2] https://docs.python.org/2/library/itertools.html

by

OPTIONS for Swift

Lately i have been working on implementing OPTIONS (1) verb for the storage nodes in Swift. The proxy server already implements this. The use of OPTIONS is to find out information on the server or other resources. In Swift’s case, OPTIONS is to return the publicly available HTTP methods of the server i.e. allowed methods, and also the server type. For this implementation, a base class was created (this was inherited by all the storage servers – Account, Container, Object) and all the public methods of the server were called by using:

inspect.getmembers(self, predicate=method)

For the returned methods from the server, the attributes ‘publicly_accessible’ is checked and added to a list and the sorted list is returned when OPTIONS is called. But these allowed methods change if the storage node cofig sets the server to be a replication server. To allow this check I extended the base class to include replication server check. The __call__(2) method was making this check earlier. I modified __call__ to access the methods from the base class.

I proceeded to add tests for it and everything went fine – almost. One of the existing Swift tests started failing. It was failing because, __call__ method is being tested by using a Magic Mock(3) method. One of the community members pointed the fact that magic mock method is callable but not a method. __call__ makes replication server check by calling the allowed methods from the base class. And the base class uses the predicate method. So it is modified to use predicate callable.

inspect.getmembers(self, predicate=callable)

That solved the issue for the test and I went on to add further tests to assert the OPTIONS functionality and also the replication server check in __call__ method.

References:

(1)OPTIONS method requests information on communication options available for a specific resource, either at the origin or an intermediary. In Swift’s case, it is at origin which is the proxy server.

(2) __call__ If this method is implemented for a class, it makes a class instance callable. For example:

class foo:
def __init__(self, a, b)
…

def __call__(self, a, b)
…

x = foo(a, b)     -> calls __init__ method
x(a, b)               -> calls __call__ method

(3) Magic Mock: unittest.mock is Python’s test library. MagicMock is a subclass of Mock with default implementations of most of the magic methods. (This is a very interesting concept and it deserves an entire post about it along with examples.)

by

OpenStack Swift

From this post to the next few ones, I will write about my current contributions to OpenStack-Swift as part of the Outreach Program by Gnome foundation.

Swift is an object storage engine written in Python. I am working to make sure clusters are set up with the correct configuration. To do so OPTIONS * verb is used to get the information about servers which then the swift client can access and make sure the configuration is done right.

Before diving into the details of the implementation, this post will brief over swift architecture.

Swift object storage allows you to store and retrieve files. It is a distributed storage platform (API accessible) for static data. Data is stored in a structured three level – account, container, object.

Account server – Account storage contains metadata descriptive information about itself and the list of containers in the account.
Container server – Container storage area has metadata about itself (the container) and the list of objects in it.
Object server – Object storage location is where the data object and its metadata will be stored.

Swift Architecture

Storage node is a machine that is running swift services. A cluster is a collection of one or more nodes. Clusters can be distributed across different regions. Swift architecture stores by default three replicas (of partitions) for durability and resilience to failure.

The Auth system:

TempAuth – In this, the authentication part can be an external system or a subsystem with in Swift. The user passes an auth token to Swift and swift validates it with the auth system (external or within). If valid, the auth system passes back an expiration date, and Swift stores the expiration part in its cache.

Keystone Auth – Swift can authenticate against OpenStack Keystone system.

Extending Auth – This can be done by writing a new wsgi middleware just like Keystone project is implementing.

Proxy server – Proxy server lets you interact with the rest of the architecture. It takes incoming requests, looks up the location of account, container or object in the ring.

The ring – A ring defines mapping between entities stored on the disk and their physical location. There are different rings for accounts, containers and one object ring per storage policy (more on this below). To determine a location of any account, container or object, we need to interact with the ring. The ring is also responsible for determining which devices are used for handoff in failure scenarios.

Storage policies – Storage policies provide a way to give different feature levels and services in the way a object is stored. For instance, some of the containers might have default 3x replication, the new containers could be using 2x replication. Once a container is created with a storage policy, all the objects in it will also be created with the same policy.

Details of these will be explored as I move ahead with my work.

Sources: A lot of my understanding and this briefing comes by reading the docs and by making contributions to the code.