Learn

Learn about latest technology

Build

Unleash your talent and coding!

Share

Let more people use it to improve!
 

Relationship between docker-cqlsh-cassandra

martes, 17 de julio de 2018


We have the most simplest architecture, two docker images/containers.  One of them with a Cassandra db and the other one contain an application(in my case Scala-Akka apps) could be any apps in any language but with ONE important exception our apps are using in his processes the CQLsh tool and specifically CQLsh tool - COPY FROM/TO.
zzz
In the above structure when the problems arise? The Application hosted in Container - II have some scripts and functions that make use of CQLsh tool for IMPORT FROM CVS file to Cassandra db(Container - I).

The main aspects in our lab:

Cassandra versions tested for us(From Official Repository)(Container - I):
  • 3.11.1
  • 3.7
  • 3.1.1
  • 3.2
CQL version: 3.4.+(Container - I)(Container - II)

When CQL version is not compatible you can add the following parameter( it is not recommended ):
  • cqlversion=Change cqlversion and use the  Database's cqlversion you want connect to
containerId=Container Id where is hosted Cassandra Database

We need to know from where is listening Cassandra (listen_address)  ref. Cassandra configuration file:
  • container_ip_address_where_is_cassandra=sudo docker inspect containerId -f '{{.NetworkSettings.IPAddress}}'
You ought define container_ip_address_where_is_cassandra, Cassandra listen_address  property as a OS Environment variable.

cqlsh --cqlversion=cql_version_compatible  listen_address

You can use the default port(9042) or any other if you change the configuration in cassandra.yaml

And the proper Docker File for CQLsh installation in our Container - II ought be:

1
2
3
4
5
6
7
8
..........
# Installing python
RUN apt-get update; \
  apt-get install -y python curl
# Installing cqlsh
RUN bash -c "python <(curl https://bootstrap.pypa.io/get-pip.py)"; \
  pip install cqlsh
...........
code. 1

The above(code. 1) is an snapshot of the Docker file part that make the cqlsh installation.

What can be done with this installation:
- create keyspaces
- create tables
- select
 
But the above installation does not encompass TWO important commands:

- COPY TO
- COPY FROM

When database is in different servers you use to get this mistake:

<stdin>:2:'module' object has no attribute 'parse_options' failing with the parse_options

I have analyze both source code(CQLsh from Cassandra and CQLsh from Python pip installation[pip in code. 1] ) and are different as well as the way that each code tackle the parse_options. You can have a look in CqlSh pip installation source code and the Cassandra cqlsh tool in your Cassandra container in  /usr/bin/cqlsh.py.

That is why if we need to make a COPY FROM/TO with CQLsh tool above solution (code. 1) It is NOT allowed.

There are several solutions for the above problem one of them is to install the CQLsh that come with Cassandra Official Image. It will be necessary only if I need to use CQLsh - COPY FROM/TO in any other case cqlsh via pip installation (code. 1) in image/CONTAINER II is good enough.

So My CQLsh Tool installation is from Cassandra Installation(ref. Cassandra versions tested):

Where the libraries in the Cassandra installation ver 3.7 - 3.11 can be found:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
# cassandra download and run cassandra image 
docker run --name cassandra_image_37 -d cassandra:3.7

# copy python libraries from cassandra container
sudo docker cp cassandra_image_37:/usr/lib/python2.7 .

# cassandra libraries from cassandra container
sudo docker cp cassandra_image_37:/usr/share/cassandra .

# cqlsh python src and phyton shell script
sudo docker cp cassandra_image_37:/usr/bin/cqlsh .
sudo docker cp cassandra_image_37:/usr/bin/cqlsh.py .

# python bash from cassandra container
sudo docker cp cassandra_image_37:/usr/bin/python2.7 .

# open ssl libraries from cassandra container
sudo docker cp cassandra_image_37:/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 .

# cripto libraries from cassandra container
sudo docker cpcassandra_image_37:/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 .

Project for create the image with java base + scala and CQLsh tools for execute COPY FROM or COPY TO can be found in my github project.

These architecture is valid only for images that are living in the same Docker. In the same way we run containers  in the same Docker which is very easy to inspect. You can include Docker Machine and config the docker-compose.yml properly if you want to work with more replicas. Another different thing is when we are talking about orchestration (swarm, kubernetes, etc) and the communication between different machines/pods.




Testing RESTful PlayFramework applications

miércoles, 30 de mayo de 2018

As was  argued by Robert C. Martin in his Clean Code bible, there are several aspects that we have to pay attention when we design our tests:
       - Minimize the number of assert per concept.
       - Test just one concept per test function.
       - Test should not depend on each other. 
       - Test should run in any environment. 
Test should pass or fail BUT you should not make process or any other thing for check if the Test was OK or failed.
Write the TEST first OR with the code, NOT after it be running in a production environment.

I am going to talk about Test Implementation in Scala and more specific in Scala  Playframework applications. If you come from Java programming language perhaps the more easy way is by ScalaTest - FlatSpec, it is a good starting point.

So when we are trying UNIT TEST in Play Framework, our minimal tests should include the followings artifacts:
  • Controller
  • Actions
  • Routes
  • Our services or functionalities, obviously
For make the previous topics more easy I have some IMPORTANT tips:
  1. Encapsulate your operations in services.
  2. Inject the classes that manage the above services.
  3. Create the relationship Class - Trait. It will let you create binding in your tests with Guice in a more easy way and at the same time get the benefits already known of this programming practice.
The main ideas are:
  • Mock services that you don't want to test. Your tests must be focused as much as you can in the functionalities that you want try.
  • Test the Controller.
  • Test the actions: When we talk about test actions that means test our action builders,  test the definitions that process every path in each request. A good idea is that action builders can process all related with our request as for example content-type. It will let us  filter for an expected Content-Type request and if it isn't what we expect stop the processing.
  • Test the routes: It is important to know that all routes defined for us are working properly or at least can be reached by the proper path.
  • Test any other Function or Service related with HTTP request. For example methods/functions related with content negotiation normally need make use of HTTP Header and in that case we will need set the header in a Fake-request.
  • Test services functionalities. I  won’t include anything related with HTTP request here. Just our own functionalities.
In an "Ideal Hello World Play Framework application" EVERY THINGS is very simple BUT the problems arise in Real World application with escenarios like that:
  • Multimodule applications.
  • Several route files.
  • Several bindings class that are injected in Singleton Class.
  • Class referencing configurations files.
  • And for example you have bindings with Class that inject other Singleton class and that Class at the same time have some references to configuration files in your Project. 
I will talk here about testing problems that are not clearly treated in Play documentation and how tackle it. When you are in real life with play framework applications is when you understand how tedious is mock a binding in Guice when you trying to create Fake apps for Testing.
It is more easy MockitoSugar in this context and could be better if when you mock any component  in Guice we could have the MockitoSugar concept instead of having to create a Well Mocked Object and binding it when you try to make an App via Guice Builder.

A design made for an easy/smart test:

I will not tell here why is so important dependency injection, you can find that information in every where even in playframework documentation.

In the following example I will show a controller that manage the actions:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
class FootballLeagueController @Inject()(action: DefaultControllerComponents, services: TDataServices) extends BaseController with ContentNegotiation {

  /**
    * Best way, using a wrapper for an action that process a POST with a JSON 
    * in the request.
    *
    * @see utilities.DefaultControllerComponents.jsonActionBuilder a reference 
    *      to JsonActionBuilder a builder of actions that process POST request
    * @return specific Result when every things is Ok so we send the status 
    *         and the comment with the specific json that show the result
    */

    def insertMatchWithCustomized = action.jsonActionBuilder{ implicit request =>
    val matchGame: Match = request.body.asJson.get.as[Match]
      processContentNegotiationForJson[Match](matchGame)
  }

// ..............

  def insertMatchGeneric = JsonAction[Match](matchReads){ implicit request =>
    val matchGame: Match = request.body.asJson.get.as[Match]
    processContentNegotiationForJson[Match](matchGame)
  }

// ..........

  def getMatchGame = action.defaultActionBuilder { implicit request =>
    val dataResults:Seq[Match] = services.modelOfMatchFootball("football.txt")
    proccessContentNegotiation[Match](dataResults)
  }

..............

}
code_example 1 ref. source code Controller

Some of my recommendations for  the real life Controllers:
  • I mostly need a custom action so inject a component [ref. code_example 1 line 1] in the previous snapshot of code [action: DefaultControllerComponents]) and this CUSTOM action will simplify the processing of your requests in Play and at the same time this design will make easy your test, something very important in our TDD process as we will see later.
  • Encapsulate the operations that must be done in your actions in services and INJECT it in your controller([ref. code_example 1 line 1[services: TDataServices]), some tips here: Inject a trait and bind that trait to a concrete class both things are truly important.  If any service include for example any customizable object like a database connection do NOT inject at this level do it in the concrete class. 
In our Controller [ref. code_example 1we are going to Test first the Controller:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
/**
  * Created by ldipotet on 23/09/17
  */
package com.ldg.play.test


import com.ldg.basecontrollers.{DefaultActionBuilder, DefaultControllerComponents, JsonActionBuilder}
.......
import com.ldg.model.Match
import com.ldg.play.baseclass.UnitSpec
import controllers.FootballLeagueController
import org.mockito.Mockito._
import org.scalatest.mock.MockitoSugar
import play.api.test.FakeRequest
import play.api.test.Helpers._
.......
import services.TDataServices


class FootballControllerSpec extends UnitSpec with MockitoSugar {

  val mockDataServices = mock[TDataServices]
  val mockDefaultControllerComponents = mock[DefaultControllerComponents]
  val mockActionBuilder = new JsonActionBuilder()
  val mockDefaultActionBuilder = new DefaultActionBuilder()
........
  val TestDefaultControllerComponents: DefaultControllerComponents = DefaultControllerComponents(mockDefaultActionBuilder,mockActionBuilder,/*messagesApi,*/langs)
  val TestFootballleagueController = new FootballLeagueController(TestDefaultControllerComponents,mockDataServices)

  /**
    *
    * Testing right response for acceptance of application/header
    * Request: plain text
    * Response: a Json
    *
    */

  "Request /GET/ with Content-Type:text/plain and application/json" should
    "return a json file Response with a 200 Code" in {

      when(mockDataServices.modelOfMatchFootball("football.txt")) thenReturn Seq(matchGame)
      val request = FakeRequest(GET, "/football/matchs")
        .withHeaders(("Accept","application/json"),("Content-Type","text/plain"))
      val result = TestFootballleagueController.getMatchGame(request)
      val resultMatchGame: Match = (contentAsJson(result) \ "message" \ 0).as[Match]
      status(result) shouldBe OK
      resultMatchGame.homeTeam.name should equal(matchGame.homeTeam.name)
      resultMatchGame.homeTeam.goals should equal(matchGame.homeTeam.goals)

    }

  /**
    *
    * Testing right response for acceptance of application/header
    * Testing  template work fine: Result of a mocked template shoulBe equal to Res
    * Request: plain text
    * Response: a CSV file
    *
    */

  "Request /GET/ with Content-Type:text/plain and txt/csv" should "return a csv file Response with a 200 Code" in {
    when(mockDataServices.modelOfMatchFootball("football.txt")) thenReturn Seq(matchGame)
    val request = FakeRequest(GET, "/football/matchs").withHeaders(("Accept","text/csv"),("Content-Type","text/plain"))    val result = TestFootballleagueController.getMatchGame(request)
    val content = views.csv.football(Seq(matchGame))
    val templateContent: String = contentAsString(content)
    val resultContent: String = contentAsString(result)
    status(result) shouldBe OK
    templateContent should equal(resultContent)
  }

  "Request /GET with Content-Type:text/plain and WRONG Accept: image/jpeg" should
    "return a json file Response with a 406 Code" in {
    val result = TestFootballleagueController.getMatchGame      .apply(FakeRequest(GET, "/").withHeaders(("Accept","image/jpeg"),("Content-Type","text/plain")))
    status(result) shouldBe NOT_ACCEPTABLE
  }
}
code_example 2 ref. source code ControllerSpec 

So if I am planning to test my Controller and if at the same time it is being injected by several component, the first thing that I need to do is give value to those components and because our target here are NOT the components I will mock them! [ref. code_example 2 line 22 - 25]:

  val mockDataServices = mock[TDataServices]
  val mockDefaultControllerComponents = mock[DefaultControllerComponents]
  val mockActionBuilder = new JsonActionBuilder()
  val mockDefaultActionBuilder = new DefaultActionBuilder()

TDataServices is a trait so we need binding it to a Concrete class, that is a very good practice. As the play specification indicates: by default Play will load any class called Module that is defined in the root package (the "app" directory) or you can define them in any place and indicate where to find it in the play configuration file(application.conf) under play.modules configuration value.

..............
class Module extends AbstractModule {
   ...................
   bind(classOf[TDataServices]).to(classOf[DataServices])
      .asEagerSingleton()
  }
}
code_example 3 ref. source code bindings

You can find more information in Play Documentation about custom and eager bindings. You should pay attention about this, it is important for several practices and in our case we will see the benefits of it in Testing.

I recommend Test with Guice only when we do not have any other choice. That is why if I need to inject a data base connection, for example, I wouldn't do it in the Controller it is better do it in the service class indeed.

Our first Test Testing Controllers begin in [ref. code_example 2  line 38 - 48] :

I need to simulate a behaviour so if any process call  services.modelOfMatchFootball[ref.code_example 1 line 27] I will return Seq(matchGame):
when(mockDataServices.modelOfMatchFootball("football.txt")) thenReturn Seq(matchGame)
in my FootballControllerSpec.
In the same way I will mock the controller because I want to inject all mocked component too and test the flow of the controller. Remember that we are in [ref. code_example 2  line 38 - 48] so I have a Fake request , something important is that we can add to this request a header, body(json, text, xml), cookies and whatever that any request has. Test will execute Controller but with mocked components and will process everything and return a response in the same way:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
.........................      
when(mockDataServices.modelOfMatchFootball("football.txt")) thenReturn Seq(matchGame)
      val request = FakeRequest(GET, "/football/matchs")
        .withHeaders(("Accept","application/json"),("Content-Type","text/plain"))
      val result = TestFootballleagueController.getMatchGame(request)
      val resultMatchGame: Match = (contentAsJson(result) \ "message" \ 0).as[Match]
      status(result) shouldBe OK
      resultMatchGame.homeTeam.name should equal(matchGame.homeTeam.name)
      resultMatchGame.homeTeam.goals should equal(matchGame.homeTeam.goals)
..........................

If  TestFootballleagueController.getMatchGame method had any parameter then the statement would be TestFootballleagueController.getMatchGame(anyparam)(request)[ref.  line 5 previous code]. In our example I have to deal with content-negotiation so I have added a header easily to my FakeRequest and then we  expect the right processing from my content-negotiation method and the right response.
It is important to know that FakeRequest(GET, "/football/matchs") is the same that FakeRequest(GET, "/"), it has nothing to do with the routes defined in the conf/routes file.

In our Controller [ref.code_example 1we are going to Test now the Actions:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
package com.ldg.play.test

import akka.stream.Materializer
import com.ldg.basecontrollers.{BaseController, DefaultActionBuilder, JsonActionBuilder}
import com.ldg.implicitconversions.ImplicitConversions.matchReads
import com.ldg.model.Match
import com.ldg.play.baseclass.UnitSpec
import play.api.test.FakeRequest
import play.api.libs.json._
import play.api.test.Helpers._
import play.api.http.Status.OK
import play.api.inject.guice.GuiceApplicationBuilder
import play.api.mvc.Results.Status
import scala.io.Source

class FootballActionsSpec extends UnitSpec {

  val jsonActionBuilder = new JsonActionBuilder()
  val defaultActionBuilder = new DefaultActionBuilder()
  val jsonGenericAction = new BaseController().JsonAction[Match](matchReads)

  val rightMatchJson = Source.fromURL(getClass.getResource("/rightmatch.json")).getLines.mkString
  val wrongMatchJson = Source.fromURL(getClass.getResource("/wrongmatch.json")).getLines.mkString

  implicit lazy val app: play.api.Application = new GuiceApplicationBuilder().configure().build()
  implicit lazy val materializer: Materializer = app.materializer

  /**
    * Test JsonActionBuilder:
    *
    * validate: content-type
    *           jsonBody must be specific Model
    *
    * @see com.ldg.basecontrollers.JsonActionBuilder
    *
    * Request: application/json
    *
    */

  "JsonActionBuilder with Content-Type:application/json and a right Json body" should "return a 200 Code" in {
    val request = FakeRequest(POST, "/")
      .withJsonBody(Json.parse(rightMatchJson))
      .withHeaders(("Content-Type", "application/json"))

    def action = jsonActionBuilder{ implicit request =>
      new Status(OK)
    }
    val result = call(action, request)
    status(result) shouldBe OK
  }

.......

  "JsonAction with Content-Type:application/json and a wrong Json body" should "return a 400 Code" in {
    val request = FakeRequest(POST, "/")
      .withJsonBody(Json.parse(wrongMatchJson))
      .withHeaders(("Content-Type", "application/json"))

    def action = jsonGenericAction{ implicit request =>
      new Status(OK)
    }
    val result = call(action, request)
    status(result) shouldBe BAD_REQUEST
  }

.......

  "DefaultActionBuilder with Content-Type:text/plain and a right Json body" should "return a 200 Code" in {
    val request = FakeRequest(GET, "/").withHeaders(("Accept","application/json"),("Content-Type", "text/plain"))

    def action = defaultActionBuilder{ implicit request =>
      new Status(OK)
    }
    val result = call(action, request)
    status(result) shouldBe OK
  }

.......

}
code_example 4 ref. source code ActionsSpec

We are going to highlight this import [ref. code_example 4  line 10]:
  • import play.api.test.Helpers._
The previous import will let call an action with an specific request(GET/POST....). An example of this call in [ref. code_example 4  line 48]:
  • call(action, request)
But the previous call need an implicit value, an instance of a Materializer so the best way for build it if we can not get an instance of our application is through Guice see [ref. code_example 4  line 25 - 26].
I am testing the Actions so in my case I will test only Action/ActionBuilder and some aspects related with the request. I won't test here the service that is executed under an specific action. Take a look about [ref. code_example 4  line 42] about our JsonBody in our FakeRequest.
It is not the objective of this post but you could take a look to JsonActionBuilder and DefaultActionBuilder [ref. code_example 4  line 19-20]

In our Controller [ref.code_example 1we are going to Test now the Routes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
package com.ldg.play.test


import apirest.Routes
import com.ldg.play.baseclass.UnitSpec
import services.{MockTDataServices, TDataServices}

import play.api.inject.bind
import play.api.test.FakeRequest
import play.api.test.Helpers._
import play.api.inject.guice.GuiceApplicationBuilder

class FootballRoutesSpec extends UnitSpec {

  val app = new GuiceApplicationBuilder()
    .overrides(bind[TDataServices].to[MockTDataServices])
    .configure("play.http.router" -> classOf[Routes].getName)
    .build()

 // implicit lazy val materializer: Materializer = app.materializer

   "Request /GET/football/PRML/matchs with Content-Type:text/plain and application/json" should    "return a json file Response with a 200 Code" in {
    val Some(result) = route(app, FakeRequest(GET, "/football/matchs")      .withHeaders(("Accept", "application/json"),("Content-Type", "text/plain")))
    status(result) shouldBe OK
  }
}
code_example 5 ref. source code RoutesSpec

This aspect is not clear in PlayFramework documentations. First of all I need to create a Fake application with the same skeleton that the real so I need to do the following:
  1. Create a fake app
  2. Mock all service that I am going to use like in my real app
  3. Create my route file
The previous 3 points are solved in  [ref. code_example 5 line 15 - line 18]:
  •  [ref. code_example 5 line 15]: We create a mock using a binding. It was one of the important thing that explain at the beginning. Mocks implemented in Guice are completely different at Mockito. In Guice you will need to mock the whole object. So what we are doing here is overriding our binding with bindings to my mocked object. You can take a look of my mocked object MockTDataServices but in general you have to give fixed values in your mocked object to any definitions in your concrete class.
  • [ref. code_example 5 line 16]: I am indicating here to my Fake app the route file that I want to use. The way to indicate to my fake app the route file is overriding the route file in our case  play.http.router -> the router class indicate the route file of our app(the routes that we want to test
I have to import the auto-generated/compiled scala class Routes. PlayFramework constructs a Route class for each configuration route file. In our Routes class generated there is an important method:

def routes: PartialFunction[RequestHeader, Handler]

The aforementioned method can give us all route in our project. So if I want to test the routes of my project then a good idea should be make every request with his specific path and test if it is working properly.

Any way this post is just the tip of the iceberg. My advice is that take a look in the test module of playframework source code that is the case ScalaFunctionalTestSpec because the documentation in this point is not pretty well on playframework documentation.

So in examples like the next snapshot of code in framework documentation


1
2
3
4
5
6
7
8
"respond to the index Action" in new App(applicationWithRouter) {
  val Some(result) = route(app, FakeRequest(GET_REQUEST, "/Bob"))

  status(result) mustEqual OK
  contentType(result) mustEqual Some("text/html")
  charset(result) mustEqual Some("utf-8")
  contentAsString(result) must include("Hello Bob")
}

They never explain how get applicationWithRouter in line 1 in the above code for instance. The solution:

1
2
3
4
5
6
7
8
  val applicationWithRouter = GuiceApplicationBuilder().appRoutes { app =>
      val Action = app.injector.instanceOf[DefaultActionBuilder]
      ({
        case ("GET", "/Bob") => Action {
          Ok("Hello Bob") as "text/html; charset=utf-8"
        }
      })
    }.build()

Testing is easy and should be an style of programming, perhaps in lightbend the make it just a bit more difficult. Anyway, in my opinion, this play framework is terrific, not your documentation.

scala in a nutshell I - Collections - List

martes, 21 de marzo de 2017

Today we will go deeper with Scala. All the code indicated in this post can be executed directly from the Scala console, so the only requirement will be the Scala installation.
In this entry we will explain in depth one of the main collections of Scala the List, today we will indicate some of the most basic aspects that we can do with this type of collection.

The best way to create variables is defining their types, as indicated below, although a less organized way would be without detailing them. In the next code segment for example, line 1 and 4 will have the same value but line 1 is much clearer:

1
2
3
4
scala> val numbers:List[Int] = List(2, 3, 4, 5, 8, 100, 200, 300)
numbers: List[Int] = List(2, 3, 4, 5, 8, 100, 200, 300)
scala> val numbers = List(2, 3, 4, 5, 8, 100, 200, 300)
numbers: List[Int] = List(2, 3, 4, 5, 8, 100, 200, 300)

The following lines of code show the creation / console result of lists of String, Int and Lists [Lists] or what are the same lists that in turn contain other lists:

1
2
3
4
5
6
7
8
scala> val cities:List[String] = List("Havana","Madrid","Oporto") 
cities: List[String] = List(Havana, Madrid, Oporto)

scala> val numbers:List[Int] = List(1,2,4,5)
numbers: List[Int] = List(1, 2, 4, 5)

scala> val listOfList:List[List[Int]]=List(List(1,2,3),List(0,1,3),List(4,2,1))
listOfList: List[List[Int]] = List(List(1, 2, 3), List(0, 1, 3), List(4, 2, 1))

Ways for make a query about if a List is empty:

1
2
3
4
5
6
7
8
scala> val emptyList = List()
emptyList: List[Nothing] = List()

scala> emptyList == Nil
res3: Boolean = true

scala> emptyList.isEmpty
res0: Boolean = true

Since many of the collection methods return a standard Scala type for optional values, we will explain this type in a  simple way and in other entries of this blog we already will make a more in-depth explanation of the same one. We speak of the Option type, it is a value to model optional values.

When a value is of type Option it indicates that it can be of two types:
  •   Some (x) where x is the current value.
  •   None which indicates that it has no value or has a null value.
In previous sections of code we have seen the simplest way to generate a list. As well as how to check if a list is empty.
A Scala list is composed of 2 parts:
  • Block of Elements
  • Nil
A simple example of the above are the next code segment, we see an empty list and then add 4 integer values, let's see the difference between the Nil and the Block of Elements:

1
2
3
4
5
scala> val newList :List[Int]= Nil
nuevaLista: List[Int] = List()

scala> 1::2::3::4::newList
res0: List[Int] = List(1, 2, 3, 4)

Those who come from other programming languages, the find method of a collection simply passes the object that we want to check if it exists in the collection, the find method of Scala, is different:

def find(p: A => Boolean): Option[A]

The find method receives as parameter a boolean expression that must satisfy an element of the collection so that it can be returned in a Some (x) where x is the found value. In case none item of the list satisfies the boolean expression, it will return a None.

1
2
3
4
5
6
scala>  val cities:List[String] = List("Havana","Madrid","Oporto") 
cities: List[String] = List(Havana, Madrid, Oporto)
scala> cities.find(city=>city.contentEquals("Havana"))
res9: Option[String] = Some(Havana)
scala> cities.find(city=>city.contentEquals("Havan1a"))
res10: Option[String] = None

The condition established in line 3 of the previous section indicates that for each element of the cities collection it will be asked if it fulfills the established condition, the first element that fulfills this condition will be returned in a Some (the_looking_element), in case that no element satisfies the condition, as is the case of line 5, will return a None.
The filter method of a collection, as its name implies, given a boolean expression will return all those elements of the collection that meet that condition. An important detail of this method is that if it finds elements that satisfy this condition, those element will be returned in a collection of the same type that we perform the filtering, in case no element meets the condition will return a collection of the type queried empty.

def filter(p: (A)  Boolean): List[A]

Let's see the following example where there is a list that satisfies the filtered condition and another one in which it does not.

1
2
3
4
5
6
scala> val cities:List[String] = List("Havana","Madrid","Oporto","Malaga","Matanza") 
cities: List[String] = List(Havana, Madrid, Oporto, Malaga, Matanza)
scala> cities.filter(city=>city.startsWith("M"))
res2: List[String] = List(Madrid, Malaga, Matanza)
scala> cities.filter(city=>city.startsWith("x"))
res3: List[String] = List()

As the following definition indicates, we have the partition method of a collection. For a boolean condition applied to a collection, all those members of the collection that meet the same are grouped on the left side of the tuple and those that do not fulfill on the right part. In short, we will have a tuple contained by two lists.

def partition(p: (A)  Boolean): (List[A], List[A])

Let's see an example about a list of integers that we want to divide in 2 groups. The elements > 20 and the rest.

1
2
3
4
scala> val numberList:List[Int] = List(1,3,4,5,8,20,28,14,12)
numberList: List[Int] = List(1, 3, 4, 5, 8, 20, 28, 14, 12)
scala> numberList.partition(number=>number>10)
res3: (List[Int], List[Int]) = (List(20, 28, 14, 12),List(1, 3, 4, 5, 8))

Let's look at our result on line 4 of the previous code segment. Now suppose we want to store those results for process them. Obviously it must be through a tuple, in each part of it we will return a list.
In the left element of the tuple will be those elements that fulfill the boolean condition, in our case the number > 20 and in the right those that do not. Once again the elements will be returned in a collection of the same type that the collections involved in the operation.

1
2
3
scala> val (elementGreaterThan10,elementLessThan10) = numberList.partition(number=>number>10)
elementGreaterThan10: List[Int] = List(20, 28, 14, 12)
elementLessThan10: List[Int] = List(1, 3, 4, 5, 8)

In the previous section we leave the result in a tuple whose members (elementGreaterThan10 and elementLessThan10) we can use later in any part of our code.

final def span(p: (A)  Boolean): (List[A], List[A])

The Span method of a collection, returns a prefix / suffix tuple, from the first element of the collection that does not meet the condition will conform the second part of the tuple, no matter that there are other remaining elements that meet the condition. The simplest way to explain it will be through an example.

scala> val listOfIntegers:List[Int] = List(15, 10, 5, 20, 12,8)
listOfIntegers: List[Int] = List(15, 10, 5, 20, 12, 8)

scala> listOfIntegers.span(_ < 20)
res12: (List[Int], List[Int]) = (List(15, 10, 5),List(20, 12, 8))

As we can see in the previous code segment, we have made a span with those elements less than 20 and has divided our list starting from the number 20(is not less than 20), the first element that did not comply with the condition. No matter that there are other elements after the number 20 that meet the condition, from the first one that fails will form the other part of the tuple. Once again the elements will be returned in a collection of the same type that the collection involved in the operation.

Next we will see the groupBy method, to apply on a collection and that has the following signature:

def groupBy[K](f: (A)  K): Map[K, List[A]]

This method Partitions the list in a map of lists according to a particular function. So for example if we group a list of String by its length all those elements of the list that have the same length will be in a same key K and will belong to the same List [A].

1
2
3
4
5
scala> val seqOfString :Seq[String] =  Seq("test", "food", "trick", "deal", "canada", "drawing")
seqOfString: Seq[String] = List(test, food, trick, deal, canada, drawing)

scala> seqOfString.groupBy(_ length)
res2: scala.collection.immutable.Map[Int,Seq[String]] = Map(5 -> List(trick), 4 -> List(test, food, deal), 7 -> List(drawing), 6 -> List(canada))

In the previous example we have a sequence of String (line 1) and in line 4 we group the items of the collection by the length of the same and we will have as result a Map [Int, List [String]] where each key K is a Length and the value associated with the key is a list that groups all elements of the same length (K). As an important detail, the key K is the type that returns our function used to discriminate, in our previous example we used the length of the string to discriminate so the key is an Integer.
In our next example imagine a list of integers in which we want to group the numbers greater than 20 in one list and the rest in another. So our result will be a Map [Boolean, List [Int]] where we will have 2 keys (K) false and true (line 5). For the key K = true we will have as value a list that groups all the items in the collection greater than 20 and for K = false those less than 20 or equal.

1
2
3
4
5
scala> val integerList: List[Int] = List(1,2,34,33,56,56,32,20)
integerList: List[Int] = List(1, 2, 34, 33, 56, 56, 32, 20)

scala> integerList.groupBy(_ > 20)
res4: scala.collection.immutable.Map[Boolean,List[Int]] = Map(false -> List(1, 2, 20), true -> List(34, 33, 56, 56, 32))

Next we will treat probably one of the most important and most used methods of collections:

def map[B](f: (A)  B): List[B]

Create a new collection by applying an specific function to each of the elements of the collection on which we want to make the map. Once again the elements will be returned in a collection of the same type that the collection involved in the operation.

1
2
3
4
5
6
7
8
scala> val listForMap:List[Int] =List(10,20,30,40,50)
listForMap: List[Int] = List(10, 20, 30, 40, 50)

scala> val multipliedList = listForMap.map(_ * 2)
multipliedList: List[Int] = List(20, 40, 60, 80, 100)

scala> val multipliedList = listForMap.map(element => element * 2)
multipliedList: List[Int] = List(20, 40, 60, 80, 100)

The previous code segment shows how a new list is created by applying a function to each of the items in the collection. In line 4 multiplies each element of the list x 2, the result of this line and the one of line 7 is the same only that in line 4 we use what we call placeholder, later we will see as depending on one utility or another It will be more convenient to use one option or another.

Let's take a more direct look at the use of functions. We will put the simplest possible example of applying functions, but it can be applied to any function. Remember that we always return a collection of the same type as the collection on which we perform the operation.

In the next code segment line 1 and line 4 are shown in two different ways to define a function that multiplies the input parameter x 2. Remember that listForMap was already defined in the previous code segment and now we will see how are applied the different functions to each elements of the collection.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
scala> def flong(x:Int):Int= x*2
flong: (x: Int)Int

scala> def  f: Int => Int = x  => x*2
f: Int => Int

scala> val multipliedList = listForMap.map(element => f(element))
multipliedList: List[Int] = List(20, 40, 60, 80, 100)

scala> val multipliedList = listForMap.map(element => flong(element))
multipliedList: List[Int] = List(20, 40, 60, 80, 100)

In next entries I will put examples of a greater complexity in the use of this method although the explained here can be extrapolated to any situation.

We will see another important method in a collection. We refer to flatMap. This method constructs a new collection by applying a function to all items in the list and using the elements of the resulting collection. In general it makes sense in nested collections, we talk about collections that contain other collections and we want our resulting collection to be the most external collection, although we may also occasionally seek the opposite or that the resulting collection is the innermost. Either way is better to see it through examples.

def flatMap[B](f: (A)  GenTraversableOnce[B]): List[B]

The following example defines a list that contains a list of cities or a list in which each one of its items is in turn another list of String. As the following example indicates:

1
2
3
4
5
scala> val ciudades: List[List[String]]=List(List("madrid","barcelona"),List("havana","cienfuegos"),List("london","manchester"))
ciudades: List[List[String]] = List(List(madrid, barcelona), List(havana, cienfuegos), List(london, manchester))

scala> ciudades.flatMap(x=>x.map(cities=>cities.toUpperCase))
res1: List[String] = List(MADRID, BARCELONA, HAVANA, CIENFUEGOS, LONDON, MANCHESTER)

We apply a function to each of the elements (cities) of the list and the result is left in a new collection. Notice how the internal collections have been processed and their elements have been taken to the container collection, it is always the same with the flatMaps, if instead of each Items being a list of String we were in front of certain object, we could act on each one of the objects and return in the container collection the elements resulting from the application of a certain function on the items in the collection.

Let's see another example to see it from another point of view:

1
2
3
4
5
scala> val listaDeString: List[String] = List("madrid","londres","paris")
listaDeStrings: List[String] = List(madrid, london, paris)

scala> listaDeStrings.flatMap(char=>char.toUpperCase)
res9: List[Char] = List(M, A, D, R, I, D, L, O, N, D, O,N, P, A, R, I, S)

We process the elements and result pull them out to a container collection. Here each of the elements will be each character that make up the array of string that form our words. Remember that we always return a collection of the same type that we perform the operation.

Now that we have seen both the flatMap and the map, let's see an example that combines both, where a List of Maps will return a List of a Map, all have been joined in the container collection (in other entries, the Map will be explained as Collection, we speak of a key-value structure in a quick way):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
scala> val seqMapNumbers: Seq[Map[String,Int]] = List(Map("Six" -> 6, "five" -> 5, "four" -> 4),Map("Nine" -> 9, "eight" -> 8, "seven" -> 7))
seqMapNumbers: Seq[Map[String,Int]] = List(Map(Six -> 6, five -> 5, four -> 4), Map(Nine -> 9, eight -> 8, seven -> 7))

scala> seqMapNumbers.flatMap((x=>x))
res3: Seq[(String, Int)] = List((Six,6), (five,5), (four,4), (Nine,9), (eight,8), (seven,7))

scala> seqMapNumbers.flatMap((x=>x)).map(y=>y._1)
res4: Seq[String] = List(Six, five, four, Nine, eight, seven)

scala> seqMapNumbers.flatMap((mapa=>mapa)).map(mapa=>mapa._1)
res5: Seq[String] = List(Six, five, four, Nine, eight, seven)

scala> seqMapNumbers.flatMap((mapa=>mapa)).map(mapa=>mapa._2)
res6: Seq[Int] = List(6, 5, 4, 9, 8, 7)

According to the previous code segment we have as input a Seq [Map [String, Int]] , a sequence where each Item is a Map and when applied flatMap on it we have a list of Tuples or what is the same a List of One Map. All items that were in individual maps have been removed and can now be processed as a common Map.

Let's look at the following code section:

1
2
3
4
5
scala> seqMapNumbers.flatMap(x=>x)
res9: Seq[(String, Int)] = List((Six,6), (five,5), (four,4), (Nine,9), (eight,8), (seven,7))

scala> seqMapNumbers.flatMap(x=>x).toMap
res8: scala.collection.immutable.Map[String,Int] = Map(four -> 4, Nine -> 9, seven -> 7, five -> 5, eight -> 8, Six -> 6)

With regard to what we explained above, we see how in line 4 applying a map to our resulting flatMap we obtain a Map of all the elements. We started with Seq [Map [String, Int]] and got Map [String, Int], in this case we have opened the content to our inner collection.

We have headOption, method that returns the first element of the collection, was used in combination with filter, so that we filter a collection and if the filtered collection we apply the headOption it returns the first element of the collection in an Some(first_element) or a None as no element meets the requirements.

def headOption: Option[A]

Imagine in the next code segment a list of integers (line 1). Now we filter the list for those elements that fulfill a certain condition and then we get the first element of the subset already filtered (line 4). As a detail, if the list is empty (line 7) and instead of a headOption we had a head method would give us an exception, this is the great advantage of our headOption.

scala> val listOfNumber: List[Int] = List(10,30,45,76,66,20)
listOfNumber: List[Int] = List(10, 30, 45, 76, 66, 20)

scala> val optionOfHeadOfNumber: Option[Int] = listOfNumber.filter(_ > 30).headOption 
optionOfHeadOfNumber: Option[Int] = Some(45)

scala> val optionOfHeadOfNumber: Option[Int] = listOfNumber.filter(_ > 100).headOption 
optionOfHeadOfNumber: Option[Int] = None

A solution similar to the previous but more complete comes with the find method.

def find(p: (A)  Boolean): Option[A]

Find the first element, if it exists and satisfies the Boolean condition. Our previous combination (filter + headOption) is done in a single method, achieving the same result and the consequent advantage of its use, in a line do almost everything.

scala> val optionOfHeadOfNumber: Option[Int] = listOfNumber.find(_ > 30)
optionOfHeadOfNumber: Option[Int] = Some(45)

Let's see the operation of the fold method which "merge or synthesize" the elements of the collection using an associative operator (which can be executed in an arbitrary order [sum, multiplication]). Always have an initial value or accumulator.


def fold[A1 >: A](z: A1)(op: (A1, A1)  A1): A1

It is important to emphasize that in the fold the operations on the elements set of the collection do not have an order and besides the accumulator (z) can be applied as parameter to the operator (op) an indeterminate amount of times, as indicated by the specification .
Let's look the following example:

scala> List(1,2,3).fold("gdg")( _.toString() + _.toString())

1st iteration -> "gdg" + "1" -> "gdg1"
2nd iteration -> "gdg1" + "2" -> "gdg12"
3rd iteration -> "gdg12" + "3" -> "gdg123"

["possible fourth iteration"] -> "gdg123" + "gdg" -> "gdg123gdg"

And so on. In all tests that I have performed the behavior has been similar until the "3rd iteration" but if we consider the Scala  language specification the result can be as indicated until the fourth iteration and even could be continue exiting iterations. In addition, the order of operations may NOT be as indicated above. That is why our accumulator is recommended to be a neutral value (0 for integer values, empty string for String values, etc).

Let's see in the following example how to "merge" or synthesize a list starting from "zero" so that our final result is the sum of all the elements of the list. Our accumulator is the basis of the function that we will apply to the collection with which as a final result we will get the sum of the elements of the list + the sum of our accumulator one or n times, just as recommended by the specification, if this element is neutral (it is the case) will not affect the final result.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
scala> val listOfInteger: List[Int] = List (1,2,3,4,5) 
listOfInteger: List[Int] = List(1, 2, 3, 4, 5)

scala> (listOfInteger fold 0) (_ * _)
res11: Int = 0

scala> (listOfInteger fold 0) (_ + _)
res12: Int = 15

scala> listOfInteger.fold (0) (_ + _)
res13: Int = 15

scala> listOfInteger.fold (10) (_ + _)

Lines 7 and 10 are two ways of expressing the same thing, which in our case is reduced to adding the elements of an integer list.

We have several details to highlight:
Always the result of the previous sum is added to the next element and our accumulator in the lines that interest us is 0, can be passed as a parameter of the operation at any point in the process. However the result of an operation like the present one in line 13 would be 25 (10 [our accumulator] + 15). In both cases (line 10 and 13) we do not obtain a collection, as the signature of the method indicates, we obtain the list "merged" or "synthesized" as a result of an associative operation (in this case sum).

Let's see the following example to leave us in a clearer way the functionality of this method.


1
2
3
4
5
scala> ((listOfInteger) fold "") ((s1, s2) => s"$s1 - $s2")
res15: Any = " - 1 - 2 - 3 - 4 - 5"

scala> ((listOfInteger) fold "test") ((s1, s2) => s"$s1 - $s2")
res16: Any = test - 1 - 2 - 3 - 4 - 5

As we can see in the previous code segment, now our accumulator is a string. See the difference when the accumulator is an empty string and when it has the value "test". We have pending the different types of data in Scala so in this moment not to worry about the type Any that appears lines 2 and 5, although they are of great importance in this case, considering the signature of the method. The fact that the accumulator (test) is not a neutral value can result in other random values different from what is shown in the previous section of code.

The following operation will generate an error because it is an operation in which if decisive the order of the operations to be performed:

scala> List("1","2","3").fold(0)(_ + _.toInt)
<console>:11: error: value toInt is not a member of Any
       List("1","2","3").fold(0)(_ + _.toInt)

It is important to note that in situations like this (where the order of operations is important) we can not consider the use of fold. Based on the previous line of code would be an error the following analysis:

1st iteration -> op (0, "1") -> 0 + "1" .toInt -> 1
2nd iteration -> op (1, "2") -> 1 + "2" .toInt -> 3
3rd iteration -> op (3, "3") -> 3 + "3" .toInt -> 6

Starting from the fact that the fold nature itself indicates that there is no set order for operations then NOTHING indicates that the first iteration is op (0, "1") -> 0 + "1" .toInt -> 1 could just be any other combination in which for example is op (0, "1") -> 1 + 0.toInt -> 1 and we would already have our first error.

We must know an important detail of the functionality of fold depending the object that apply this method will not be the same the concept of "synthesize or merge" in all types of collections or objects on which fold method can be applied, so we will assume here that the concept of polymorphism is known by all.

Some of the above errors can be solved with our next method.

def foldLeft[B](z: B)(op: (B, A)  B): B

As noted in the specification op (... op (z, x_1), x_2, ..., x_n) we will apply the operator in a direction from LEFt -> RIGht
So that from LEFt to RIGht:

1st iteration op (accumulator, first_element_element) -> Result_1
2nd iteration op (Result_1, second_element_element_collection) -> Result_2
3rd iteration op (Result_2, third_element_element_collection) -> Result_3

And so on until the collection ends.

One of the big differences between foldLeft and fold is that in fold there is no guarantee on the order in which the elements are processed while in foldLeft we know that we start from LEFt -> RIGht.

Let's look at a case study where we use foldLeft and however fold method can not be used:

scala> List("1","2","3").foldLeft(0)(_ + _.toInt)
res59: Int = 6

The sequence order of the operations executed here is as follows:

1st iteration (0 + "1" .toInt) -> 1
2nd iteration (1 + "2" .toInt) -> 3
3rd iteration (3 + "3" .toInt) -> 6 [the last result]

We will always have the guarantee that any foldLeft we make on a collection the first operation applied on the collection will be op (accumulator, 1st_collection_item).

Our next method will be the foldRight whose operation, speaking in the simplest way will be similar to foldLeft but both the iterations and the order of the initial accumulator will be contrary to foldLeft, we will start from right to left..

def foldRight[B](z: B)(op: (A, B)  B): B

As we can appreciate the signature of the method is the same although the way to do the iterations is RIGht -> LEFt

We have a very simple example but that will give us an idea:

scala> List(1,2,3,4,5,6,7,8,9,10).foldRight(1)((a,b) => {println(s"$a,$b");a})
10,1
9,10
8,9
7,8
6,7
5,6
4,5
3,4
2,3
1,2
res12: Int = 1

Let's look at another similar example(see foldLeft section):

scala> List("1","2","3").foldRight(0)(  _.toInt + _)
res14: Int = 6

We see how has been necessary to change the order of the parameters. And now let's see the order of the operations.

Important: See the order or position that the accumulator (on the right) occupies in our case.

1st iteration op (first_element_element, accumulator) -> Result_1
2nd iteration op (second_element_element, Result_1) -> Result_2
3rd iteration op (third_element_element_, Result_2) -> Result_3

A common operation of all folds is that if the collection is empty we return the accumulator, which in general we must try to be NEUTRAL.

What we mean by NEUTRAL:
If it's a list -> Nil
If it is a multiplication operator -> 1
If it is a String operator -> ""

And so on.

If we can adapt our code to use fold would be perfect although many times we will not be able to. If we can adapt our code instead of using foldRight using foldLeft it would be much better(foldLeft on foldRight). We will explain it in future entries.

We will now deal the reduce method, whose main difference with the fold method is that it does not have an initial value, so we have to make sure that the collection on which we use it is not an empty collection. Also here is "reduced or merged" a collection using associative operators.

def reduce[A1 >: A](op: (A1, A1)  A1): A1

Let's take some example like when we want to generate composite csv files obviously for several CSV entries. Given the following collection let's see how foldLeft resolves generate a file whose lines have values separated by commas:

scala> val values: List[String] = List("ValueCSV1", "ValueCSV2", "ValueCVS3","ValueCSV4","ValueCSV5")
values: List[String] = List(ValueCSV1, ValueCSV2, ValueCVS3, ValueCSV4, ValueCSV5)

scala> values.fold("")(_ + "," + _)
res11: String = ,ValueCSV1,ValueCSV2,ValueCVS3,ValueCSV4,ValueCSV5

If we process our fold method  we can delete the "," but if we do not want to do this processing we could run a reduce method on the collection:

scala> values.reduce(_ + "," + _)
res12: String = ValueCSV1,ValueCSV2,ValueCVS3,ValueCSV4,ValueCSV5

scala> values.reduceLeft(_ + "," + _)
res13: String = ValueCSV1,ValueCSV2,ValueCVS3,ValueCSV4,ValueCSV5

All you have to guarantee is that values are not an empty list. In those occasions that reduceLeft and reduceRight methods lead us to the same result the most advisable is to use reduce, the most common situation. As we explained previously for fold method, when we need the order of the operations to be a specific order we will have to use the appropriate (left or right) reduce method.

scala> listNumber.reduce(_ min _)
res31: Int = 2

scala> listNumber.reduce(_ max _)
res32: Int = 20

scala> listNumber.reduceLeft(_ max _)
res33: Int = 20

scala> listNumber.reduceRight(_ max _)
res34: Int = 20

scala> listNumber.max
res35: Int = 20

scala> listNumber.min

This is a case where it would be better to use reduce or neither that. But imagine that our op instead of being max or min, is a division, in this case we must care about the order of the parameters and we would then have to see what reduces method (left or right) to use.

In this entry we have described the most important methods of List, the rest of them you can find in the api of Scala language, just mention methods to recommend that are well explained in the api:

Find, flatten, diff, contains, exists FilterNot, foreach, last, tail, collect and above all arithmetic operations on list (::, :::, +:, ++:, ++ and rest of this kinds of op)

Anyway, before doing any operation better look at the Api, in most of the times there are always different routes, some more optimal than others.