Testing Tableau and automating it with Wiiisdom Ops for Tableau

0:00When was the last time you thought about

0:01testing when building your Tableau work

0:03books?

0:03In today's world, what we typically do is

0:05we build them in desktop or in WebEdit.

0:08We then publish them up to the server.

0:10We use them like a user would, and we

0:12essentially open that up to the business

0:13that they too can see those checks.

0:16Once that's happened, we sort of call that

0:18quality assurance.

0:19We move it maybe to the appropriate project

0:21and it goes public.

0:22That is our testing process.

0:24Now, if you're in an enterprise setup, you

0:25definitely do a lot more than this.

0:26You maybe have dedicated testing teams, but

0:29one of the big issues with all of this is

0:32that it's manual.

0:33It's actually quite time intensive.

0:34If you add up all the combined time of

0:36everyone involved, it's actually quite a

0:38tough amount of time to swallow.

0:40And the other thing here is it's not very

0:42easily repeatable.

0:43Let's say the team changes, someone else

0:45builds a workbook.

0:46You might all have different testing

0:47approaches and you might not actually be

0:49testing the same thing.

0:51Now, what if you could test a workbook

0:53based on a set of requirements?

0:55And more importantly, you could automate it

0:57.

0:57That's exactly what I'm going to be

0:58covering in today's video, where I'm going

1:00to check out a tool that helps you do this.

1:02Let's get stuck in.

1:03Now, if you're a testing aficionado, skip

1:04ahead to the next chapter.

1:05You probably know all this already, but I

1:07thought I'd covered the basics of testing

1:09and what it is exactly.

1:10We test along with some testing terminology

1:12.

1:12So let's start with the first thing.

1:14When you test a workbook, what you

1:16typically want to do is test its

1:17functionality.

1:18You want to test the filters, parameters

1:20work.

1:20You want to make sure that they've been set

1:22to the correct values.

1:23You want to make sure that all the values

1:24appear in the filter that should be there.

1:26You also want to do a few things like click

1:28on a few items, make sure everything is in

1:30the right place.

1:31But you might also have some guidelines.

1:32Examples are corporate guidelines that tell

1:34you how to build a dashboard, where to put

1:36the logo, how much space to leave on the

1:38left hand side panel, where to put filters.

1:40These are all things that some

1:42organizations do clearly sort of want you

1:44to do every single time you build a

1:46dashboard.

1:47This is typically known as functional

1:48testing.

1:49You're testing that the technical aspects

1:51of the dashboard are set up and work the

1:54way they should.

1:55Now, the next type of test you might do is

1:56something called regression testing, and

1:58this is actually very similar to cross

2:00environment testing apart from one thing.

2:02In regression testing, what you're doing is

2:04you're taking a snapshot of a workbook.

2:06You're testing its performance and its

2:07capability.

2:08Once you've taken that snapshot, you're

2:10comparing all future tests against that

2:12baseline, but you're keeping it on the same

2:14environment.

2:15So an example is you might publish

2:16something to production.

2:18This might be something that everyone's

2:19using.

2:20And then from that point onwards, you're

2:21basically testing against that version,

2:23that snapshot.

2:24You're testing the performance, you're

2:26testing the capability.

2:27And what you're looking for is a regression

2:29in its performance.

2:30And that allows you to then go in and

2:32troubleshoot and see if something has

2:34caused that or see if the data set has

2:36changed to cause that or even see if your

2:38whole environment has changed.

2:39And that's what's causing the issues.

2:41It's a very common type of test.

2:43Now, cross environment testing is actually

2:45very similar, except for this time you're

2:47testing between two different environments.

2:49Here you're testing maybe between your dev

2:51and your production sites on Tableau server

2:53, which in some cases actually runs on the

2:55same hardware.

2:56If that's not the case, then this is

2:57probably going to make more relevance

2:59because you might have resource constraints

3:01on dev versus production if you're doing

3:03them as physically separate servers.

3:05Or if you're running them on the same

3:07server, you might have resource limitation,

3:09which is new in 22.1 that you might be

3:11working around.

3:11So essentially, you just want to make sure

3:13that everything is working as it should do

3:15between two environments.

3:16The last type of testing you might want to

3:18do is something called performance testing.

3:19Now, in the Tableau world, performance

3:21testing is something that happens all the

3:23time, but there is sometimes a sort of a

3:25mixed methodology and ways of doing this.

3:27In this particular case, we're talking

3:29about the performance that the end user

3:31sees.

3:31So is the workbook fast to load?

3:34Is it responsive when they select the

3:35filter?

3:36Does it change the dashboard quickly and

3:38efficiently?

3:38What are the issues to do with the server

3:40infrastructure that's causing these issues?

3:42Performance testing will pick these out and

3:44enable you to sort of drill down into what

3:47exact part of the pipeline is causing these

3:49issues.

3:49That's pretty much all the testing types

3:51that we do in the Tableau world.

3:52There's more if you go to other sort of

3:54technologies and industries, but that's

3:55pretty much it in a nutshell.

3:57OK, now that we've covered that, we're

3:58ready to get stuck into this tool.

4:00But before I do, I just want to make a

4:01couple of things clear.

4:03The tool I'm going to be looking at today

4:04is called WisdomOps.

4:05Now, they reached out to me a long time ago

4:07. It's actually been my bad for not getting

4:09to this video a lot sooner than I have done

4:11.

4:11Life got in the way and I just wasn't able

4:13to make this video in the standard that I

4:15wanted in a good time.

4:17Now, the second thing to bear in mind here

4:19is that they've not sponsored me to make

4:20this video.

4:21They've not sort of asked me for a favour

4:22of a review.

4:23This is generally my impression of using

4:25this tool.

4:26That said, I'm also not a testing expert.

4:28I'm not someone who spends a lot of time

4:30testing when I typically go into clients.

4:32I'm a consultant, so I tend to hand over my

4:34work to someone else who does the testing.

4:37So I also don't have the sort of thorough

4:39testing experience that I should have in

4:41order to be able to review this tool.

4:43So this is also not a review.

4:44It's purely my impressions of working with

4:46a tool like this.

4:47And there'll be a lot of opinion and

4:49commentary, but please don't take that as

4:51fact or as any sort of statement of what

4:53you should think about the product.

4:55Now, the very last thing is that they gave

4:57me access to a trial period.

4:59Now, anyone can get a trial period of up to

5:0114 days, but they actually gave me an

5:03extended trial so that I could work on this

5:05video a little bit better.

5:06So that's pretty much the extent of the

5:07support they've given me.

5:08Now that that's all clear and out of the

5:10way, let's hop into the tool and check it

5:12out.

5:13OK, so here we are. I'm actually using

5:14Windows for this video.

5:16I could have used the Mac version, but I

5:17actually had an issue with the way that my

5:19Mac is set up and the specific version of

5:21Chrome that I'm using.

5:22So to avoid any sort of issues relating to

5:24Chrome being funny, I switched over to

5:27Windows.

5:28Now, this is the tool itself. It actually

5:30has a very familiar interface.

5:31You can sort of see they've borrowed a lot

5:33of hints from Tableau itself with sort of

5:36this left pane on the left hand side, of

5:38course, and a discover pane on the right,

5:40which sort of links you off to Tableau

5:41resources.

5:42I think that's very deliberate just to try

5:43and make the tool feel familiar.

5:45Now, when you land on the tool, you do get

5:47an extra explanation of some of the testing

5:49types I talked about before.

5:50You've got the different types of testing.

5:52But in essence, what you have to do is you

5:54have to open a project.

5:55Now, a project is think of it like a set of

5:57tests that you're going to run.

5:59So in this case, I'm actually going to open

6:01a project and this is going to hopefully

6:03open a Dropbox folder.

6:04Now, I've put this on Dropbox because this

6:06is where I want to sync all my tests

6:07between the two computers.

6:09So I tested it on Windows, then I'm

6:11switching over to Mac.

6:12And so I wanted to test everything in one

6:13place.

6:14So now I just select the project and select

6:16open.

6:17And when this opens, it should load up all

6:18the tests that I've run in the past.

6:20And when we do that, we actually get a

6:22summary.

6:22Now, this summary just gives us an overview

6:24of everything that we've run already.

6:26You can see that some have failed, some

6:28have succeeded and some have never been run

6:29.

6:30You can also see the total number of runs

6:32in each particular case.

6:33But in order to set up a test, there's sort

6:35of a process you need to go through and it

6:37's actually quite a simple one.

6:38The first thing is you need to define a

6:40context.

6:40The context just basically tells the tool

6:43where this test is going to run.

6:45So let me go over here to the top right

6:47hand side and you can see that I actually

6:49have the ability to define different places

6:51where the test can run.

6:52So these are essentially just connections.

6:54Think of them as that.

6:55And you can set one as the active context

6:57depending on the text you're trying to run.

6:59So I'll go back to this one here.

7:01In particular, you can see that I'm

7:02connected to my Tableau Online instance.

7:04This one is also connected to Tableau

7:05Online, but in a slightly different way.

7:07And this one is actually connected to a

7:09Tableau server directly.

7:11And again, it's connected in a slightly

7:12different way.

7:13So all of those are contexts and I can

7:14switch between those contexts depending on

7:17the test or the type of test that I'm doing

7:20.

7:20Once you've set that, you can of course set

7:21any one of these to be the active context

7:23when the tests run.

7:25And that allows you to essentially switch

7:26between profiles or switch between

7:28environments very easily and quickly

7:30without having to constantly log in and do

7:31whatever you need to do.

7:32So I'll leave this at the moment on this

7:34one because it's actually what I'm going to

7:36be using for this test.

7:38Now, the other thing you'll see here is

7:39that these tests, I've named them in a very

7:41sort of basic way.

7:42But the way this tool works is actually

7:44quite smart.

7:45Essentially, this interface is just a user

7:48interface that actually controls a command

7:50line tool.

7:51I'll put an image up that Wisdom have on

7:53their website about the architecture of

7:55this platform.

7:56But in essence, it's just basically

7:58allowing you to build a set of tests which

8:00are stored as JSON object.

8:02JSON means JavaScript object notation.

8:04It's a simple way of storing data in web

8:06friendly format that then allows you to

8:08pass that off to another application that

8:10can then take that process and do something

8:12with it.

8:13Essentially, that information is passed on

8:15to the command line and the command line

8:17actually runs a range of different tools.

8:19The other thing to be aware of is that

8:20Wisdom is running a technology called Sel

8:22enium.

8:22Now, Selenium is just a clever way of

8:24running a browser and controlling it.

8:27So in testing, you're of course going to be

8:29wanting to click on filters, go to

8:31parameters and change them.

8:33This is all this tool is doing.

8:34It's building a layer on top of that tool

8:36to allow you to do those things in a

8:38programmatic way and in a repeatable way.

8:40The very final thing is that some of the

8:42tools that Wisdom are using are actually

8:44tools that you should already be familiar

8:45with if you're used to testing with Tableau

8:48.

8:48A good example is Tabjolt.

8:49That's actually what they're using to do

8:51the performance testing when they go down

8:52to the performance testing settings, which

8:54we'll go through very shortly.

8:55Check out the timestamp if you want to jump

8:57ahead to that now.

8:58So now that we've set the context of how

9:00the tool works at a very, very, very basic

9:02level, let's actually look at some of these

9:04tests and see how they work.

9:05Let me go over here to the left hand side.

9:07You can see that once I loaded my project,

9:08I actually opened up all of these same

9:10tests we can see here in the summary.

9:12And if I just simply go to the functional

9:14test, you can see that we have a workflow

9:16like environment.

9:17So you can essentially add these tasks and

9:19these tasks happen in chronological order.

9:22So if I'm going to build a test, I'm going

9:24to first need to log into Tableau.

9:26I'm going to need to open a visualization.

9:28I'm going to want to set the filter range

9:30to a specific set of values.

9:32You can see here that I've set it to the 1

9:34st of January 2018.

9:35I might want to then check that some

9:37filters have specific values and you can

9:39see that that's all been set up here.

9:41So then now that that's all set up, you can

9:43do a very simple thing.

9:44You can run the test and you just need to

9:46make sure that you set the right context.

9:48In this case, the context is my Tableau

9:50server.

9:51So that's what I'm going to do.

9:52I'm going to hit the run test button just

9:54so you can see it working.

9:55And when you do that, a couple of things

9:57happen.

9:57You can see the command line interface at

9:59the very bottom.

10:00It's essentially firing up the tools that

10:01it's going to be using.

10:03And it's kind of scary because Chrome opens

10:05up and it's completely automated and it's

10:07essentially controlling the browser.

10:09Chrome is actually saying that, look, I'm

10:10being controlled by a tool.

10:12Now, once it's doing that, it actually then

10:14starts carrying out the test.

10:16The first thing it will always do is log in

10:17, make sure you're you, and then it will go

10:19to the visualization and it will carry out

10:21the filtering.

10:22If I move my face here to the left, we'll

10:23hopefully see that it's doing that.

10:25You can see that I maybe interrupted it in

10:27the process, but now I managed to finish

10:29that process just fine.

10:30And now that the test is done, it's

10:31obviously complete and we're pretty much

10:33good to go.

10:34You can see here on the bottom left that it

10:36says, OK, the testing was complete.

10:38Success is for partial success is zero, sk

10:40ips at zero, failures zero.

10:42So that's the test run. Very, very simple.

10:44Now, the great thing is I can, of course,

10:46repeat this test.

10:47So if I just want to keep running this test

10:48, this is great.

10:49It's going to be making it very easy for me

10:51to just see how everything is working.

10:54And I can do this over time and it will

10:55just repeat the same test again and again

10:57and again.

10:58And this is just so much faster.

11:00In fact, I think this is doing it faster

11:02that I could manually do this myself.

11:04Imagine doing all these steps.

11:05It would take me considerably longer

11:06because it's not automated.

11:08So this is great.

11:09This just sort of proves the simplicity and

11:11the repeatability of the tool and how it

11:13works.

11:14If you're new to testing, this is probably

11:15blowing your mind.

11:16But if you're familiar with testing, this

11:18is probably really boring.

11:19So let's dig into what specific features

11:22this tool has for doing this case

11:24functional tests.

11:26So over here on the left, you can see that

11:28I have a range of different tasks.

11:29And what I've tried to do is not dive into

11:31these too much, but I'm just going to cover

11:33the different sort of types of tests you

11:35can do with these.

11:35You can of course see that you've got some

11:37of the ones I've already talked about.

11:38Log into Tableau, open the visualization.

11:40You can also drive the browser and you can

11:42also have another version which is drive

11:44the

11:44browser, the single sign on.

11:46This is specifically useful when you're

11:47using single sign on in an enterprise

11:49context.

11:50And there's a lot of customizations there

11:51that allow you to make sure that the

11:53interface

11:53is going to work as it should do.

11:55The other thing, you've got filter checks

11:57here.

11:57So you can set the date range.

11:58You can check the list.

11:59You can set the range in general.

12:01If you keep going down, you can also do

12:03things like set the parameters, which is

12:04actually

12:05quite a useful thing to do.

12:07Now you then have this big body of tools

12:09called assertions.

12:10And assertions essentially allow you to

12:12check particular behavior and make sure

12:14that the

12:14data that comes out of it actually matches

12:16something else that you already know the

12:18value

12:18to.

12:19So this is a really good way to go in and

12:21programmatically check exact results.

12:23And you can do this a couple of ways.

12:24You can do this by just using a filter, but

12:26you can even go down and do this and check

12:28this against the data warehouse.

12:29You can see here the assert SQL equals

12:32allows you to essentially take a SQL

12:33statement that

12:34you've run against the data warehouse and

12:36you can check the workbook against that SQL

12:38statement.

12:39So is the analyst building the workbook

12:41getting the exact same result as a SQL

12:43statement run

12:43against the warehouse directly?

12:45That's a simple, very powerful test you can

12:47do.

12:48You can do this on something as simple as

12:49the number of records to make sure that the

12:51database has been refreshed and that the

12:53workbook has actually picked it up.

12:54That's a very simple way of testing that.

12:57And then you can have the outcome of that

12:58sort of go off onto other processes and use

13:00things like the rest API to make sure that

13:03those things are working as expected.

13:05Now this is a really sort of powerful set

13:07of tools, but again, I'm not an expert.

13:08So I highly encourage you to go out to

13:10wisdom's website and they also have a

13:12couple of YouTube

13:13videos that you can check out the dig into

13:15this a little bit deeper.

13:16If you want me to dig into these deeper in

13:18a deeper way, then let me know in the

13:19comments

13:19below and I'll spend a little bit more time

13:21going into the nooks and crannies of each

13:23and every one of these tools at some point

13:25in the future.

13:26Now that's the functional test.

13:27You can do a lot with that.

13:28You can build a whole bunch of different

13:30sort of workflows and get that to pretty

13:32much work

13:33the way you want it to work.

13:35Let's look at the next type of test, which

13:37is going to be a performance and stress

13:38test.

13:39Now this is actually quite simple.

13:40When you go to add a task, you've only got

13:43one option, the performance test.

13:45When I click on this, you can see that it

13:47actually asked me a couple of very basic

13:48things.

13:49Number one, what type of test are you doing

13:51?

13:51It's a performance test.

13:52You can give it a name.

13:53Number two, how long do you want it to run

13:55for?

13:55Number three, the concurrent clients.

13:58And in this particular case, what it's

13:59essentially asking is how many people do

14:01you want me to

14:01simulate?

14:02Do you want me to behave as one person or

14:04do you want me to behave as a hundred

14:07people?

14:07You can do all of that.

14:08And essentially what this is doing is

14:10building a test and it's using the test

14:13capability

14:13of Tabjolt.

14:14So these are the variables you'd actually

14:17put into Tabjolt to get that to work.

14:19The other thing you might then do is give

14:20it a username and password that it can use

14:22to log in and then do the test.

14:24And so when I do this test, this is

14:25actually going to take a while.

14:26So I'll speed this up in post so you don't

14:28have to wait a minute for it to run.

14:30But when you run this test again, it's

14:31going to use my same context and it's going

14:33to open

14:33up the browser and do exactly the same

14:35thing.

14:36And as it's doing that, it's essentially

14:38just running the test in the background.

14:40Some of these tests can be done completely

14:42through the command line interface.

14:44You don't have to have this interface open

14:46up at all.

14:47But when this test finishes, what we'll do

14:49is we have a look at the results and we'll

14:50see what it thinks about the performance of

14:53the server that I'm testing.

14:56Okay, so you can see that the test

14:59succeeded, it finished, everything's run

15:04and it actually

15:06writes a report.

15:07You can just see that it's labeled here.

15:08It writes most of its reports in HTML and

15:10it stores them in a specific place where

15:12you've

15:12set up the project.

15:14But you can also look at these reports

15:15yourself.

15:15Now, if I just go back to this task

15:17properties, when I set up the basic test,

15:20one thing I

15:20forgot to mention is that you can actually

15:22build in SLAs.

15:23SLAs are just a way of basically saying,

15:25look, this is the minimum level performance

15:27we'd

15:27expect.

15:28When you're doing a test like this, you don

15:30't do it over 60 seconds, you do it over a

15:31long

15:32period of time.

15:33And what you want to do is set some

15:34conditions.

15:35So in this case, I've set a performance

15:37goal of 10 seconds.

15:38Ten seconds tends to be, in my mind, it's a

15:41pretty, pretty generous performance.

15:44Maybe I would say three seconds if I was

15:46using a modern web applications.

15:47But in the Tableau Analytics world, you're

15:49just going to call it 10 seconds for now.

15:51The availability goal is look, 99.5% of the

15:54time, it was actually able to carry out the

15:56test.

15:57So that's my sort of, you know, me testing

15:59how many nines of availability I want with

16:02this particular workbook.

16:03But what you're really testing is the

16:05server.

16:05Then the aptX goal.

16:06Now the aptX goal is something that was

16:08really new to me.

16:09aptX is essentially a way of standardizing

16:11the way you measure performance and how

16:13users

16:13experience that performance.

16:15And there's a wonderful video that I'll

16:17link to in the description below.

16:19And it goes into this in a lot more detail

16:21that I can squeeze into this video here.

16:23But in essence, it's actually a really good

16:24way of standardizing performance in your

16:26organization,

16:27given the two variables we've already

16:29talked about, which is performance and

16:30availability.

16:31You can also then exclude certain things

16:33from the test if you want to from this

16:34particular

16:35view.

16:36But let's check out the report that it

16:38created from the run that I just did.

16:39So you can see that I've actually run this

16:41test three times.

16:42And each of those times, it keeps the

16:44result.

16:44So I can actually track this over a period

16:47of time.

16:47If I open up the report success for the

16:49most recent run, I can actually look at

16:51this in

16:51the full screen view and it opens up this

16:53setup.

16:54Now here you can see a general overview of

16:56what's going on.

16:57You've got the artifact, which was

16:58essentially the test, the JavaScript object

17:01notation that

17:01was used to run this test.

17:03And then you've got the aptX score.

17:04This is what it shows you first, because of

17:06course, this is the industry standard item.

17:08And what you can see here is that each of

17:10these things called a one which is

17:11excellent.

17:12Now if I go to response times, I can scroll

17:15down and I can see here that look of all

17:17the

17:17interactions pretty much everything loaded

17:19in that 10 second time limit.

17:21You can see here the line that has been

17:23drawn for that 10 seconds.

17:25You can see the time, but you can also see

17:27a dot for each and every interaction.

17:29So it's actually repeating these

17:30interactions multiple times.

17:32And you can even see that there's a little

17:33bit of variance.

17:34For example, the interaction with the viz

17:36in this particular case was slightly slower

17:38.

17:38So that's not a bad thing.

17:39This happens if you do things a thousand

17:41times.

17:42None of those times something else is going

17:44to cause an issue.

17:45But it gives you this information very

17:47easily.

17:47Now the availability is pretty

17:48straightforward, because I only run this

17:50for 60 seconds.

17:51Of course, it's going to be up there for a

17:54long time.

17:54And you've got all the different sampling

17:56and information and sessions ID that it was

17:58basically collecting.

18:00So this is sort of the granular information

18:02of every single thing that it was doing,

18:04all

18:04the small sort of habits and behaviors that

18:06it was running in the background.

18:08So this is a really nice way of doing this.

18:10Now you're probably wondering, well, this

18:11is great data, but I don't want it here.

18:13I want it in a database.

18:14Well, that's another thing you can do.

18:16If you go to the integrations, you can

18:18connect it up to all of these tools, email,

18:21Slack,

18:21webhooks, Postgres, and roll bar.

18:23Now I'm not too familiar with all of these.

18:26But what I do know is that the Postgres

18:28database here is going to be handy, because

18:30if instead

18:30of storing these as files or HTML files,

18:32you probably want to be piping these into a

18:34database.

18:35So you have a history of all these tests.

18:37And it's stored in a way that allows you to

18:39then do analytics on that data in a more

18:41sort

18:42of resilient way.

18:43I like the webhooks, and I like Slack and

18:45email, because of course, it's nice to know

18:46that the test has run and it's completed.

18:49Or it's also nice to be notified that a

18:50test has failed, which is probably the

18:52thing that's

18:52more pertinent.

18:53The Slack integration is going to allow you

18:55to send things to Slack to say exactly the

18:57same.

18:58So depending on how you work, whether you

18:59've gotten rid of email and you've used

19:00Slack

19:00instead, all of these things are going to

19:02be great.

19:03And of course, the webhook will allow you

19:05to hopefully use this thing with Teams and

19:07other tools like that as well.

19:09So all of that functionality is built in

19:11there.

19:11It's a really sort of nice to use tool.

19:13So that's the performance test in a very

19:15simple nutshell.

19:17I think it's really nice to have this here.

19:19Now one of the interesting things I noticed

19:21with this particular tool, it is actually

19:23just building a tabjolt test for you.

19:25If we go down here into the command line,

19:28you can see here that the created the tree

19:30successfully using tabjolt, and it's using

19:33tabjolt2022.1.

19:35And it's essentially creating something

19:37called a test plan.

19:38When you build tabjolt test, you are

19:40actually building it using this sort of

19:42technology.

19:43And so this is actually quite nice to know

19:44and see because if you use tabjolt and you

19:46trust tabjolt, this is giving, I think more

19:49people an easier front end to that

19:51capability

19:52rather than just making it something that

19:54only the server admin can do, which I think

19:56is a nice to have.

19:57Now the next thing I want to look at is

19:58regression testing.

19:59So let's go back and click on the

20:01regression test.

20:02So now you can see here that I previously

20:04run a regression test.

20:05Let's go back to task and just see how that

20:07is set up.

20:07Obviously here you can see that I have two

20:09options.

20:10I can sign in with single sign on.

20:11This is what essentially this option does.

20:13And then you've got the regression test

20:15here as well.

20:16And when you click on the regression test,

20:17what you all, all you're doing is you're

20:19basically

20:19going to get a visualization and you're

20:21going to check it.

20:21And I'm going to take you through the

20:23process of doing that by essentially ret

20:25aking the

20:25baseline for this exact visualization.

20:27So here I have a test called regression

20:29test, and I'm going to go and get a

20:31specific visualization.

20:33And in this case, the visualization has

20:35seven views.

20:36How do we take a snapshot?

20:37Well, over here on the right hand side, you

20:39hit baseline and it'll open this view.

20:41Now this view is basically going to say,

20:43look, this is what I already have as my

20:45baseline.

20:46What would you like me to do?

20:47Do you want to refresh it?

20:48Do you want to update the current selection

20:50?

20:50Maybe there's a new tab that's been added.

20:51So you want it to go and check if there's

20:53anything new.

20:54All of this is possible.

20:55You can also delete specific things from

20:56this snapshot.

20:57So let's just go ahead and refresh the

20:59snapshot so you see a genuine experience of

21:01how this

21:01works.

21:02So what it will do is it will go through

21:04using my variables again, log me in, go to

21:06the visualization,

21:07make sure it's got an understanding of that

21:09.

21:09Then it will load this view.

21:10This view then just starts scanning the

21:12dashboard.

21:13It essentially loads the dashboard up in

21:15lots of different sort of ways, and it

21:17checks various

21:18things on those dashboards.

21:19Now those things it checks are based on a

21:21set of variables that you set up when you

21:22do the regression testing, and it takes a

21:25snapshot of what those things look like.

21:27Now another useful thing here is that this

21:29is also going to be useful in instances,

21:31maybe

21:31where you want to see if something has

21:33changed when it shouldn't have changed.

21:34Maybe someone has removed the logo.

21:36Maybe someone hasn't actually updated

21:38something and you'd like a standardized way

21:40of testing

21:40that.

21:41This is going to help you do that.

21:42So I'll let this scan finish.

21:44It takes a little while because it has to

21:45go through each tab.

21:46You can see that it's simulating its way

21:48through that.

21:49And when it's done, we'll have a look and

21:51see how the regression test actually runs.

21:53Okay, so you can see that the refresh has

22:02happened.

22:04It's taken a snapshot of everything else.

22:06Now, of course, you can select each and

22:07every one of these.

22:08And when you do, it gives you a screenshot.

22:10It checks everything that's there.

22:11I can actually click on each and every one

22:13of these and get a really thorough

22:15breakdown

22:15of everything that it's found.

22:17It's looking at the columns, it's looking

22:19at the filters.

22:19I can also go ahead and look at the data

22:21inside of that.

22:22So it's actually taking a pretty thorough

22:25snapshot of what's going on.

22:26And it's persisting that in its database of

22:29information.

22:30And then when you run a regression test

22:31against this stuff, it's actually going to

22:33check it,

22:34which is actually a really, really good

22:35thing.

22:36Now, it does this for each and every works

22:37heet, each and every view.

22:39So you can, of course, make sure that

22:41everything is working as expected.

22:42So here on the left, we're going through

22:44the worksheet.

22:45Here on the top, we're going through the

22:46different views.

22:47So obviously, as you change through these,

22:48you'll get different metrics coming through

22:50.

22:50Obviously, when you click on these, you'll

22:52see that update.

22:53Now, the other thing you can do is you can

22:54obviously say to this tool, "Okay, we're

22:56done.

22:56These snapshots all look good.

22:58Let's hit OK."

22:59And we'll basically go back to this view.

23:01And now it has that snapshot, you can run

23:04the test.

23:04And the test allows you to check these

23:06various things.

23:07So you can refresh data on open, use

23:09formatted values, check columns.

23:11You can even check an image, essentially

23:13allowing you to check if a logo is missing,

23:16or you can use it to check against another

23:18image.

23:19In essence, what this is doing is when it

23:21takes that snapshot, when you actually look

23:23at the snapshot,

23:24let's open up this baseline, and let's go

23:26look at a specific view, you will see that

23:28it loads up this image.

23:29And it's actually going to check that image

23:31.

23:31Let's say you move the filter to the right-

23:33hand side, it's actually going to be able to

23:34notice that and say,

23:35"Hey, look, this looks different.

23:36This is what's new about it."

23:38So that's a pretty powerful feature for

23:40making sure that everything is exactly how

23:42it should be.

23:43There's a couple of other checks you can do

23:45, check whether parameters exist,

23:46check to whether filters exist, and all the

23:48related things you'd want to do with that.

23:49But essentially, once you've done all that,

23:51you can just go ahead, hit Run Test.

23:53It's going to ask you to save and run,

23:54which is fine.

23:55And when you save and run, now it's taking

23:57that baseline and it's going back to that

23:59visualization,

24:00and it's checking against the baseline.

24:02So in this case, nothing should have

24:03changed because I've just literally taken

24:04the snapshot.

24:05But I just wanted to show you the mechanic

24:07of that working.

24:08And at least for this part of the test, the

24:10regression test looks identical to the

24:12baseline that we took before.

24:14It's essentially going through the same

24:15process when it checks it, but it's

24:17checking against a different item.

24:19It's checking against itself rather than

24:20against what it finds on the server, which

24:23is a really nice touch, I think.

24:25Okay, so you can see here that it's

24:26finished and the test succeeded and

24:28everything's looking good.

24:30I can obviously go to the report.

24:31I can click on the report.

24:32We can go even full screen here, and you've

24:34got a breakdown of the test itself.

24:37So if we click on this, you can see that,

24:39look, Superstore test.

24:40This is the actual whole workbook, and you

24:42can see all the individual tests that were

24:45done in this particular tab.

24:46I can go to the forecast tab and exactly

24:48the same thing happens.

24:49So this is, again, just a really nice way

24:51of breaking down the test, making sure that

24:53all this stuff is visible and everything

24:55you've asked it to check has actually

24:57performed well.

24:57Again, you can export this out to the

24:59database and keep that there for history.

25:02But it actually keeps its own history over

25:03here on the left-hand side, as you can see.

25:05I think this is a nice touch.

25:07Now, the one thing I want to do is actually

25:09walk you through setting up a test.

25:10I've gone through all the different tests.

25:12We've done a functional test, performance

25:14and regression test.

25:15The cross-environment test is actually very

25:17similar to regression test.

25:18The only difference when you set this up is

25:20if you go to the task and look at this, you

25:22can obviously do a multi-site comparison.

25:24So you might have something on one Tableau

25:26site and something else on another Tableau

25:28site.

25:29And it's essentially just comparing the two

25:30and comparing the performance.

25:32You get a checklist of the things you can

25:33check.

25:34Very similar to the regression test, but

25:36slightly different because it's looking

25:37across different environments.

25:39You can obviously specify the target sites

25:41as well and then run the test to compare

25:43the two.

25:44And there you go.

25:44You have your test results.

25:46Very similar to regression test.

25:47So I won't run that now.

25:48But what I will do is we'll try and set

25:51this functional test up from scratch so you

25:53can see the exact process of doing this.

25:55So let me go ahead, delete everything from

25:57this view, and let's build a test together.

25:59Okay, so we're here.

26:00We've got absolutely nothing in this page

26:02and we're going to set up a functional test

26:03.

26:04Now, it gives you a few hints on the page

26:06about how to go about doing this,

26:08especially if you're logging into something

26:09like Tableau Online or if you're using

26:11single sign-on and you need to do a little

26:13bit more.

26:13So what we will do is we'll go ahead and

26:15just log into Tableau because I'm connected

26:17to a Tableau server.

26:18This is super simple.

26:19And we'll just say log into Tableau and it

26:21will pretty much go away and do that.

26:23Now, you might want to run this already

26:25just to test that it works.

26:26And so you can do that.

26:27But I have a lot of confidence that this

26:29works, so we're not going to worry too much

26:31about it.

26:31Now, the next thing is to open the

26:33visualization.

26:33So let's go ahead and bring this in.

26:35When we do that, of course, it doesn't know

26:37the visualization.

26:38So it goes red here just to tell us, "Hey,

26:40you need to pick a visualization."

26:41And this is actually what I want to show

26:42you.

26:43Let's go ahead and select search.

26:44You don't have to know the visualization

26:46off the top of the head.

26:47You can search for it.

26:49You can also browse the full server

26:50hierarchy.

26:51So here you can see that there's a demo

26:53site.

26:53And if I open this dropdown, every single

26:55project that's on the site comes up.

26:57And if I go into any one of these projects,

26:59all the data sources and all the workbooks

27:02actually come up.

27:03So it's a massive list of stuff.

27:04If you're using this to browse an entire

27:06server, you don't want to have to be

27:08browsing that entire thing.

27:09This is actually a private Tableau server,

27:10so that's why this is sort of faded out.

27:12Hopefully, you can still see what I'm doing

27:14in this sort of blurry mess.

27:15But nonetheless, you can also search.

27:18So I can search for something, very simple,

27:21Superstore test.

27:22And when I actually expand this, it only

27:24shows me the places where that search has

27:26actually met something.

27:27So a much, much smaller set of items

27:29because the search is being applied.

27:32And I can do a couple of things.

27:33I can tick the whole entire workbook.

27:35And what that does, if you notice, it

27:37actually means it's going to pick seven

27:40views.

27:40Now, if I expand this workbook, I can

27:43actually just pick the specific views I

27:45want to test.

27:45Because I want to test just the overview

27:48tab, just to make sure that the filters are

27:50there,

27:50I'm going to untick these and go through

27:53the rest of it.

27:54Now, the thing you might do if you tick

27:57everything is you might want to loop

27:59through various tests.

28:00So let's say I go through customers,

28:02forecast, overview, and performance.

28:03And I want to check whether a parameter

28:05exists in all of them.

28:06In that case, what I do is actually tick

28:08all of them, go and select the actual test

28:11to check if the parameter exists

28:13and that the value has been set correctly.

28:15And what it actually does is it builds out

28:17a loop.

28:17And it essentially goes through each one of

28:19these sheets to check that that's worked.

28:21And then you're pretty much good to go.

28:23But in this case, we'll just pick one view

28:25or click OK.

28:26And now that we have the tablet URL, it

28:29knows what the visualization is,

28:30we can actually start to do some things to

28:32it.

28:32So previously, what I did is I set up a

28:34filter.

28:34So I'm going to set up a date range.

28:36And I know that on this particular

28:38visualization is an order date filled.

28:40Now, you don't have to know this either.

28:42This tool is actually quite smart.

28:44You can go and tell it to, hey, go take a

28:46snapshot of the workbook.

28:48So we'll go to the workbook and allow you

28:50to take a snapshot.

28:52Now, one thing I can't figure out is in

28:54this particular step, I seem to have to

28:57reselect the tool.

29:02And I think this might just be one of those

29:04quirks to do with testing tools,

29:05where actually you have to be specific each

29:07and every time.

29:08What I'd love it to do is to pass the

29:10variables from the previous tool into this

29:13tool.

29:13So then I don't have to set this up again.

29:17If I was to, for example, select forecast

29:19here, this could lead to a failed test,

29:21because of course, I'm selecting a

29:23different view to what I previously

29:24selected.

29:25So when I click on overview, you see you do

29:27get a snapshot over there.

29:28And it's going to ask the snapshot to go

29:30and grab the filters, parameters, the image

29:32,

29:32all the summary data, all the columns as

29:35well.

29:35Click OK.

29:36And it will go off and get that.

29:38And it does that again by simply running

29:40Selenium,

29:40logging in as you go into the workbook and

29:42grabbing that information.

29:44So there's a lot of opening and closing the

29:46browser that you do nothing.

29:48It's a weird experience when you do this

29:49for the first time and the computers just

29:51control

29:51in the browser for you.

29:53It's actually something that will probably

29:54cause security concerns in some

29:56organizations,

29:56but testing software sort of goes through

29:59very thorough testing.

30:00So I'm sure that organizations that do use

30:03this have happily sort of approved this

30:05software

30:05and don't find an issue with it.

30:07Selenium itself is incredibly common in the

30:08technology space as well.

30:10And as you can see, it's taken the snapshot

30:12.

30:12Now, if I go to a specific sheet, these are

30:15all the sheets I can say, look on the sales

30:17map,

30:17I want you to filter in a specific way.

30:20Now, the date filter name, if I go to

30:22search that, you'll see that because it's

30:24taken the

30:25snapshot already, it already allows me to

30:27go and filter specific things.

30:29So I'll go select the order date.

30:31And once the order date is in there, I can

30:33tell it that, look, this is a static filter

30:35.

30:35So I'm just going to give it to date.

30:37I could give it a dynamic date.

30:38So I could say, hey, this is the relative

30:42number of seconds between these two filters

30:44.

30:44Now, that is a very sort of huge number to

30:48be dealing with if you're trying to count

30:49the number of seconds in a week.

30:51So that to me, I wish you could sort of get

30:54different units of time here.

30:55That would be a sort of a nice to see thing

30:57.

30:57But nonetheless, we'll just go with static

31:00and I'll just say 01012019.

31:03And we'll have this go to 01052019.

31:09So we'll essentially travel.

31:10I'm in the UK, so I'm using UK dates here.

31:13Where we go day, month, year.

31:14I don't know who would do anything else.

31:16But now that that's pretty much, it's

31:19actually not correctly set.

31:20If you notice here, we also need to be

31:22specific about time.

31:23And I guess this is in instances where you

31:26might also be filtering date and time.

31:28But as far as the tool is concerned, it's

31:31going to feed this in.

31:32And because there's no date time field, it

31:35's just going to keep the date in this

31:36particular thing.

31:37So again, I'd love this sort of quirk to be

31:39fixed, where if I don't enter anything,

31:41it just gives me a 00 rather than giving an

31:43error to enter a valid date.

31:45So now that that's done, we're pretty much

31:46good today.

31:47I'm going to save this just in case

31:48something happens and I lose this.

31:50And you can see that it saves the job.

31:51And it's essentially just building a

31:53JavaScript object notation.

31:55If you go over here to the right hand side,

31:57you can actually see that test.

31:58Now the thing I like about this is that it

32:00's actually quite a predictable way of

32:02writing

32:02this JSON object.

32:04And because of that, what you can actually

32:06do is you can write a tool to build this

32:07JSON objects

32:08for you.

32:09So instead of going in here and building

32:12the tasks, you can actually sort of reverse

32:13engineer

32:14some of the common tasks you do and

32:16dynamically fill this information in,

32:18depending on what

32:19you're trying to test.

32:20The benefit of doing that is you can then

32:22do really complex things.

32:24For example, you might take a tablet work

32:26book,

32:26you can open up the XML file for that work

32:28book, look at all the things that are in

32:30that workbook,

32:30and then create a test run or test script

32:33based on what's in the workbook.

32:35So you could actually write something that

32:36tests every single filter, every single

32:38parameter

32:39to a range of values by looking simply at

32:42the workbook.

32:43So I'm not sure if someone's ever tried

32:45that, but I know it's absolutely possible

32:47because

32:48this sort of setup looks very similar to

32:51the XML setup that you get inside of a work

32:53book.

32:53And all the information is just inside of a

32:55workbook.

32:55So maybe that's a new tool that maybe

32:57someone can build a tool that builds test

33:00plans based

33:00on the workbook.

33:01Maybe Wisdom can take that feedback on and

33:03build something like that.

33:05Now we've done that test.

33:07Now one thing I want to also go and do is

33:09go and see if the filter has specific

33:13values.

33:13And so what I can do is I can go and use

33:16one of these assertion and you want to

33:18basically

33:19go and check that the specific values exist

33:22in the filter pane.

33:23So I'm going to go ahead and select this.

33:26Now because we've taken a snapshot, it's

33:27actually got the snapshot ready to get.

33:29So we'll click OK.

33:30And what I should actually do is to select

33:33the specific sheet.

33:35So let's go and select.

33:37Let's do sales map and we'll go to search

33:42and you'll see the sales map has a region

33:43filter

33:44and it's a categorical list, which is good.

33:47And what I'm going to ask it to do is to

33:49check the values and the values I expect to

33:51see

33:52are north and we just scroll down here,

33:56south and central.

34:00So do all of these values exist?

34:02OK, so now that we've done that, the test

34:04is pretty much good to go.

34:06I'll go ahead and save it again.

34:07And now I think we're ready to run this.

34:10Now if this fails, it's probably because of

34:11my error here.

34:12So let's go ahead and run the test and we

34:14'll see it run through the entire test.

34:16Now let's see if I've done this properly.

34:18So we should see it open up the browser.

34:21It will first log in and then having logged

34:24in, what we should see is it open up the

34:27test

34:28window that essentially loads up the

34:29visualization.

34:30Now this is going to happen very, very

34:31quickly.

34:32So I'm going to put myself on the top right

34:33hand side here.

34:34We're going to see it filter over here on

34:36the right hand side.

34:37You can see that filter worked and it

34:39actually went and checked the values faster

34:41than I

34:41could blink.

34:42And we know that here because we go down

34:44here, you can see invoking a cert filter

34:46equals

34:47task.

34:47OK.

34:48And it's basically checked everything fine.

34:50Now what was interesting is in a previous

34:53test, I actually got a readout that

34:56specified

34:57that these were the right fields.

34:58So I just want to check that this is

35:00correct.

35:01So list categorical, check values, enable

35:05to use formative values in your filters.

35:07That is exactly correct.

35:09And this is, yeah, that's just the name.

35:11So I think it has done what I expected to

35:13do.

35:14I'm probably not using this tool to the

35:16best of its capability as well.

35:18But again, I just thought it was a really

35:20interesting thing to explore.

35:21Now in using this tool, I've had this tool

35:24now for a while.

35:25I've actually used it probably over a total

35:27of, you could say probably over the course

35:30of an entire day.

35:31I've been in it for that period of time.

35:33In order to use it, I actually got a demo

35:35from Wisdom themselves.

35:36They should have walked me through how it

35:37works.

35:38I had a really good scripted demo.

35:39I can't fault that whatsoever.

35:41And it's pretty straightforward and I

35:42really, really like it.

35:43One of the things that I don't have context

35:45for is what other tools are like.

35:47And for the record, there are other tools

35:49that do this.

35:50Wisdom is not the only player in this

35:52market, specifically working with Tableau.

35:54The other competition is also just NoTool.

35:57Doing what you've done today is Wisdom easy

36:00to use in some of these tools?

36:01And broadly speaking, I'd say yes, it is.

36:03I think one of the tougher parts of using

36:05this tool though, is just understanding its

36:07quirks.

36:08And maybe this is because I'm quite used to

36:10Tableau and I'm quite used to sort of

36:11modern

36:11tools.

36:12There are a few quirks.

36:13You saw me sort of talk a bit about them.

36:15So for example, the date range, if you don

36:17't sort of fill this information out and it

36:19just

36:20stays with the dash, you could get an error

36:22where the test doesn't run.

36:23So once you know these things, once you've

36:25sort of worked with this for a while, you

36:27start to pick up these habits.

36:28The other thing that I think I found very

36:30difficult and I haven't really shown you in

36:32this video is connecting to Tableau Online.

36:34And this is sort of caused partly by Table

36:37au, but also because I think there's a lot

36:38of

36:39change going on in the Tableau space when

36:41it comes to APIs.

36:43In essence, I struggled to get Tableau

36:45Online working when I had two-factor

36:47authentication

36:48enabled.

36:49And Tableau enabled that right at the point

36:50that I was testing this.

36:52And so I don't have an honest perception of

36:54how this tool works with Tableau Online in

36:57a two-factor environment.

36:58I think it is important to do that because

37:00in essence, you're going to need to be able

37:02to work with one of the biggest parts of

37:04Tableau in order to use this tool.

37:06But for the record, it's not that I don't

37:08think that it can't connect.

37:10You can see here that I actually have an

37:12online connection here to their test site.

37:14And when I do the test connection,

37:15everything works absolutely fine.

37:18And we can actually go ahead and build a

37:20new test.

37:20Let's go ahead and build a new functional

37:23test.

37:23And we'll call this Testing Tableau Online.

37:28And we'll click Add.

37:30This will give us a blank canvas.

37:33We'll go over here and switch my variable

37:36and set my Tableau Online test instance

37:38with

37:39wisdom as the default context.

37:41So we'll go in, log in to Tableau, and we

37:44'll open a visualization.

37:46Let's go ahead and search for that

37:47visualization.

37:48You can see here that I'm connecting to

37:50Tableau Online.

37:51It's absolutely found that, and it's worked

37:52exactly as you'd expect.

37:54We'll go and grab Superstore, and we'll

37:56grab the same overview tab that we've been

37:57grabbing.

37:58We'll click OK, and that's pretty much our

38:00visualization ready to go.

38:02And so it's not impossible to connect to

38:04Tableau Online.

38:05It's technically possible.

38:06What I couldn't debug is why my specific

38:09environment on Tableau Online just wasn't

38:11able to work

38:11with this.

38:12It also doesn't help that I have a Tableau

38:14Online instance that's run by the data dev

38:16team at Tableau.

38:18So my version of Tableau Online is actually

38:21a beta instance, so it doesn't quite match

38:23the generic version of Tableau Online that

38:25everyone uses in a production setup.

38:28So it's specifically for testing, so it's

38:30not going to work like the real one.

38:31And I've come across common bugs, actually,

38:33with that specific instance that don't

38:36exist

38:36in the main instance of Tableau Online.

38:38And so it's been actually really hard to

38:39prove.

38:39So that's definitely something to go and

38:41check out.

38:42Now, I know that Tableau and Wisdom

38:44probably don't have this issue elsewhere,

38:46because if

38:47we go over to the Wisdom website, you can

38:49see here they've actually got a really nice

38:50white paper that I was reading before I

38:51made this video.

38:52They have some good documentation on the

38:55different testing types.

38:57But if we actually go to Wisdom, let's just

38:59go type in their website.

39:02We can actually take a look at some of

39:03their customers and who they have using.

39:05And you have to sort of scroll down a

39:06little bit.

39:07If I actually go to the very top, I can

39:09look at products, and we go to Wisdom Ops,

39:12which

39:12is specifically what's being used in this

39:14specific instance.

39:16If we scroll down, we do get a very brief

39:20run through of some of their larger

39:22customers,

39:22I'm assuming.

39:23So let's keep going down.

39:24It actually works across multiple tools.

39:26So it's not just Tableau.

39:28Also works with Power BI and SAP Business

39:30Objects, which is, I think, where the

39:32business

39:32actually started.

39:33But it's being used at all these

39:34organizations.

39:35And I'm not sure if it's been used in

39:37context of Tableau itself.

39:38You'd have to ask these organizations

39:39themselves.

39:40I don't have a good picture of that.

39:42But I do know that Wisdom have a very

39:43active user group.

39:44They have lots of customers across the

39:46world that are using this.

39:47And in a Tableau space, they're actually

39:49definitely one of the more popular

39:51customers.

39:52There's a couple of other players, but it's

39:53pretty much Wisdom and one other player, I

39:55can't remember their name, who use this

39:57tool.

39:58So that's pretty much it for me.

40:00I don't want to sort of carry on too much

40:02longer without some specific guidance from

40:04you guys on what kind of tests you want to

40:06see me run in this.

40:07If you want to see me run anything at all.

40:09I'll also kind of judge the feedback based

40:11on the response of the video.

40:13I think this is one of those videos that

40:14most people are going to find when they

40:15Google

40:16testing or when they've been asked to go

40:17and investigate testing with Tableau.

40:19You come across the video and hopefully I

40:20've helped you sort of think about testing

40:22in

40:22a better way.

40:23More importantly, I've hopefully shown you

40:25what is out there in terms of tools that

40:27you

40:27can use to automate and improve the way you

40:29test your workbooks.

40:31As ever, if you've made it this far, you're

40:33an absolute fan.

40:34So thank you for watching.

40:35If you've made it this far and you're not

40:36subscribed, though, please be sure to hit

40:39the subscribe button.

40:40Over 80% of people who watch my videos don

40:43't actually subscribe, which is absolutely

40:45fine.

40:45I don't mind you getting the value from

40:47this whatsoever in whatever way.

40:48I'm here to help, but you can help me out a

40:50lot by just hitting the subscribe button.

40:52So you find out about new videos as soon as

40:54they go live.

40:55Thanks for watching and I'll catch you in

41:00the next video.