An Old Kaleidoscope Test Plan

12/25/201829 Min Read — In Career

This is old and musty and I had to clip some things out but you might find it interesting. Shipping Kaleidoscope updates and then working on V2 was so much fun and I had the best team around. It's hard to read without wincing a bit. I've gotten a lot better at my job since then.

❚❚❚ KS2

Test doc & thought guide for Hoth: Our beloved difftool Kaleidoscope sports a new scope and now helps you merge.

❚❚❚ Document Usage Guide

Tested Items
  • Internal alphas and external betas of Kaleidoscope 2.0 for Mac, internally known as project Hoth. Testing also covers helper app ksdiff and in-app purchasing.
Testable Features at a Glance
Not To Be Tested Features & Methods
  • Purchasing through the web store—outside of Kaleidoscope—is not covered.
  • This document does not attempt to go anywhere near unit testing.
  • We only approach the basics of SCM and terminal applications in regards to integration with ksdiff. No esoteric terminal shells covered.
Approach
  • We attempt to cover integration, regression, load, endurance, installation, compatibility, smoke, alpha, usability, security testing in this document as much as we are able to do so manually. Our approach is somewhere between exploratory and ad hoc
  • While the feature set can be quickly and efficiently be tested by hands and eyes, any existing automated tools or code will be noted. If you can get UIRecorder to do anything useful, please drop everything and share with the class this very instant.
  • Sure there is black, grey and white box testing but this is our Kaleidoscope, we're doing multi-colored prismatic tube testing. Get into it.
Pass/Fail Criteria
  • You are the judge, if something doesn't fail outright but seems odd or questionable, please file a case and note your findings. This document makes a lot of assumptions and won't hold your hand through each area of testing.
  • If the thought "Should I file a case on this?" comes into your head, the answer is always Yes; let the team decide what to do with the information.
Assumptions
  • As previously mentioned, some test items will have specific reproducible steps/actions but most will not. Some will have expected outcomes and others won't. That's where your lovely brain comes in. It is assumed that you will know things such as: pressing the red orb in the top left corner should shut the window, Command+Z should undo the last action, a preference toggle should be respected on relaunches, etc. If you feel like you're having difficulty approaching a thing, ask around.
Suspension Criteria
  • Any issues you find should be filed immediately, and if they are so heinous as to stop you in your testing tracks, file the case as a P1/blocker and sound the alarm.
  • You must reassess whether or not you can continue testing or start over with another configuration. Many times you ail only need to move to another setup. If you are truly blocked, alert the authorities. All of them.
Responsibilities
  • Assigned developers & designers are constantly filing new cases as part of their workflow and project management, please keep yourself apprised. project:"Kaleidoscope" -viewedby:"me" -area:"Support" OrderBy:LastEdited in FogBugz should do the trick
  • Internal Alpha releases are disbursed to the entire BPXL team and they are encouraged to use the application and report any bugs they find in their normal usage.
  • External Beta releases of Hoth are scheduled for release on October 23rd 2012.
  • Generally speaking, BPXL's Quality Assurance group is departmentally responsible for testing all our in-house products with one team member being the lead for a particular project.
  • For this document you will be asked by a team member to work through it or parts thereof.
Schedule
  • Best case scenario would be weekly run through for the features hit list, coinciding with any dev-published builds.
  • The rest is nutso-smasho. Aim high, friends.
  • Cases are due immediately — filing cases on old builds is like putting a twenty spot in the paper shredder.
  • Please report the concerned once you have finished, we'll throw a party.
  • If you can't finish the document or prescribed section, run in to any blockers or otherwise are unable to proceed, please let your team know what's up.
Deliverables
  • As you work through this document, please file your cases as swiftly as you are able. The case content should have some indicator of proposed Severity to assist triage status.
  • Assign to the project default for whatever is in FogBugz at that time and alert any team members who you think should be alerted.
  • The results of this document should mostly take the form of filed cases: Do not bring issues you find to chatrooms, phone calls, staff meetings, mail lists, scrum or the like UNTIL you have a case filed with which you can reference. I'll go as far as saying you should begin every inquiry with a link to the case. gtfo://00000
  • You are also expected to update this document as the project progresses and evolves. It's a living document and under version control. Don't file bugs on the test doc, just fix them.
Pre-testing tasks
  • Get to know the project

  • Know your roots: Pluck through the support mail in FogBugz (and the Gmail account if you can track it down)

  • Know your roots: read the transfer docs in the Dropbox kaleidoscope-transition-notes
  • Know your roots: go through the original pre-release testing doc from Sofa with a fresh install of KS1
  • Know your roots: eyeball KS1's release note history
  • Release notes should be provided before pre-release testing, otherwise you should be following along with FogBugz, SCM commits, chatrooms, mail lists, Dropbox activity and internal alphas to get a sense of where things at, yes that is a lot of noise, just focus on areas if it's overwhelming.
  • Familiarize yourself with the project basics
  • Keep abreast of existing cases by reviewing them and getting to know our known issues.
  • Obtain Hoth build
  • Grab a build from Dropbox Products/Kaleidoscope/builds/ You'll need to be running signed and proper builds in order to provide useful results. The best way to do that is used published builds from the team.
  • The next best place is Jenkins you will need to be at the MFR or on the VPN in order to access that project.
  • The last—and least best—option is to run a build you have made yourself; be aware that you may have inconclusive results when you've gone rogue as it were. Read this on manual Hoth builds

  • Install & configure supported SCM applications

  • bzr

  • Install IDE & SCM GUI apps
  • P4V

  • Files

  • grab the Ned Pixel KS2 license

  • KaleidoscopeTests git repo [on github, has submodules]
  • ask for Palladium access [my personal stash of shit]
  • Don't forget to bring a towel.
Testing Environment

Default to hardware, but go ahead and give 'er a run on VMs, just know that you will get bugs that don't pop up elsewhere and ImageScope will not work

  • Device coverage

    • Retina
    • Non retina
  • Supported Mac OS X releases

    • 10.7.3+
    • 10.8.0+
    • 10.9+
  • Binary type

    • MAS
    • Direct Sale
    • Manual Debug builds [please only use when branch hopping for team]
  • Account coverage [Integration, Registration and Installation]

    • admin
    • nonadmin
    • guest
For a fresh install
  • dump Kaleidoscope files from your machine

  • Kaleidoscope binaries on disk, including build directories and trash and Dropbox!

  • you should be using Unique build locales, go to there

  • ksdiff and friends

    • /usr/local/bin/ksdiff
    • /usr/local/bin/ksdiff-wrapper
    • /usr/local/bin/ksdiff-svnwrapper
    • /usr/local/bin/ksdiff-finder ?
  • sandboxen files in Containers

    • ~/Library/Containers/com.blackpixel.kaleidoscope
    • com.blackpixel.kaleidoscope.SandboxedPersistentURLs.LSSharedFileList.plist
  • local prefs

    • defaults delete com.blackpixel.kaleidoscope
    • com.blackpixel.kaleidoscope.plist
  • support files

    • ~/Library/Application\ Support/Kaleidoscope
  • Launch services

    • scrub it
  • Other temp stuff

    • ~/Library/Caches/Kaleidoscope
    • ~/Library/Caches/com.blackpixel.kaleidoscope
    • ~/Library/Logs/Kaleidoscope
    • private/var/folders -- you're gonna hafta look around for com.blackpixel.kaleidoscope stuff
  • Consider the following approaches

    • new user accounts
    • virtual machines
    • guest login
    • pristine/known state bootable clones
For a dirty install
  • double check that your settings and set up in general won't interfere with your testing and let 'er rip
    • that means eyeballing all of the Clean Install stuff above
  • don't forget about your integration settings
    • local to your user and also the machine
  • dont' forget about anything funny in your bash profile
    • where does git point to, for example
For the love of Gord
  • Pick the OS/release/hardware configs to be tested before you begin and then note those config details somewhere. TextExpander it. Jot it down. Make it a part of your vows. Just log it; you will get interrupted and lose track.
  • Beyond the steps noted in cases, please jab and prod all aspects of all interfaces: verify that things look and behave the way they should, especially after resizing and clicking around in things repeatedly, scroll up, scroll down, select and click, move in and out of full screen, poke around!
  • Intermittently, verify that all main menu items, contextual menu items, and shortcuts work during various workflows. Sprinkle 'em in like pepper.
  • Go extra thoroughly through all new features and interfaces but keep an eye out for regressions. Get in touch with it, stop thinking, let things happen, and be the ball.

❚❚❚ Testing KS

Dev-supplied Scenarios

Follow each one of these specific testing scenarios. These are straight from our dev team and crucial to assess. Every flight begins with a fall.

  • Testing_File_Change_Updating
  • Testing_Scenarios
  • Kaleidoscope_File_Difference_Representations
  • Merge_SHA_Examples.html

Claims Testing

These are the features listed in KS1's public marketing site and the internal Hoth Top Level Features for KS2. We are specifically checking that each thing is true.

KS1 Features From Marketing Site
GENERAL
  • Single window workflow 86'd
    • One tab per comparison
  • Compare two+ files
    • Add any number of files per tab
  • Always up to date
    • Auto-refreshes when files change
  • File Shelf
    • Quickly go from file to file
  • Path Bar
    • See where your files are
  • Keyboard shortcuts
    • Every action has a shortcut actually true now
TEXT SCOPE
  • Supports any text file
    • Plain text, source code, HTML, and import text from .doc and .rtf files
  • Clear visualization
    • Added, deleted and changed text
  • Three layouts
    • Blocks, Fluid & Unified
  • Customizable
    • Pick your font and color scheme
  • Instant search

  • Change Stepper

    • Quickly jump from change to change
IMAGE SCOPE
  • All popular image file formats
    • JPEG, TIFF, PNG, PSD and more.
  • Four connected layouts
    • Two-Up, One-Up, Split & Difference
  • Comprehensive image support
    • RGB/CMYK/LAB, alpha, any size.
  • Multi-touch support
    • Smooth scrolling and pinch to zoom
  • Customizable environment
    • Change background color, difference mask and checkerboard settings
  • Two difference mask modes
    • Spot absolute or relative changes
INTEGRATION
  • Git, Mercurial, SVN & Bazaar
    • Integrate with a couple of clicks
  • ksdiff Command-line tool
    • Integrate with practically anything
  • Changeset
    • All your changed project files conveniently grouped in one tab
  • Third Party Apps
    • Versions, Gitbox, Tower, TextMate's SVN and Mercurial bundles and more
  • System Requirements
    • An Intel based Mac
    • Mac OS X v10.5.8 or v10.6.2 or later 86'd
KS2 Top Level Features

Marketing materials in which to test have not been nailed down yet. Reference project details as you are able.

  • All of KS1 features
  • Full Screen Support (KS1 carry over that wasn't on the site)
  • Mac OS X Lion & Mountain Lion / 64bit
  • Sandboxed (MAS)
  • Gatekeepered
  • Folder Compare
  • Folder Merge
  • Text Merge
  • Text Editing
  • Undo support for Merging in each scope
  • VCS Merge Conflict Resolution
  • VCS Merge Conflict Resolution for images (git & hg)
  • FastSpring purchasing/licensing
  • Possible PDF support?
  • Perforce support?

Core Values Testing

What is our mission statement? What are some test cases that will check we are in line with that? We don't seem to have one published so below are just some aspects of good software and Kaleidoscope in general. This is purely my opinion, yours may vary.

Does Kaleidoscope 2…

  • surprise and delight? have an element of fun?
  • not pester or confuse the customer?
  • get out of the way and let the customer do her job?
  • respect the customer's time?
  • understand the tasks customers will be performing?
  • have a personality and does it show through in all areas?

KS1 Regression Testing

KS2 can't be less performant than KS1. Strike a baseline and perform basic text diffing and changeset invoking with KS1 and continually test against KS2. Any slip is an emergency. Log some basics from various machines. Monitor specs and performance. Focus on these areas:

  • Complex text diffs timing and results
  • Image diff performance by view type
  • Changesets performance, across network and by VCS type

Risk Testing

Any time we merge text or files or folder, resolve a conflict, or update hierarchies we are at a huge risk of data loss. The UD part of CRUD. The product has been written to help users save themselves from data loss however, each time a users data is at risk, we need to make extra-extra sure there are no defects that would lead a user to lose dem bits. These areas of are of particular concern.

Backing out of a text merge
  • attempt to yank the rug out from under a text merge
Backing out of a folder file merge
  • attempt to pull connectivity of drive during edit
Backing out of a VCS conflict resolution
  • kill app, kill vcd wait, kill wc, underwrite wc
Losing connection when accessing data
  • attempt to pull connectivity of drive during edit

  • attempt to pull connectivity of drive during save

File contents changed by outside process
  • pull a wc for example
Folder contents changed by outside process
  • pull a wc for example
Troubles saving to disk
  • full disk
  • write protected
  • local network
  • remote network
  • moving file types between differently formatted drives
  • save interrupts
  • save to broken disk
  • saving and writing by outside application

The warning comes in two flavors, an alert (popup) and a sheet. The alert / popup should be shown only when quitting the app while there are folder merges in progress, if there are no dirty documents open.

The sheet should be shown: 1) when quitting the app with folder merges in progress, if there are dirty documents open. 2) When closing a tab that is doing a folder merge. 3) When closing a window that contains a document that is doing a folder merge (using the main menu or the stoplights).

If the sheet is showing and the folder merge completes, the sheet should go away, and the user's original action (closing a document or window) should e continued. This does not apply to the alert / popup.

The "continue copying" / "continue" option should put the user back where they were, aborting the termination or close operation. The user can then wait until the merge completes.

The "stop" option will continue immediately with the user's original request (terminate or close window / tab), after stopping the folder merge process.

Testing this bug is much easier if folder merges take a long time, so I added a flag SLOW_FOLDER_MERGE to KSFolderScopeMergeController.m. If that flag is changed from 0 to 1, folder merges will always take at least 10 seconds.

Friendly Integration Testing

Playing nice with others. Does KS work with, and can it work better with, popular text editing, IDE, etc, applications?

  • BBEdit
  • Coda
  • Cornerstone
  • Dreamweaver
  • Dterm
  • Eclipse
  • Espresso
  • Finder
  • Fish
  • Forklift 2
  • Gitbox
  • GitCola
  • GitHub for Mac
  • Gitty
  • Gitx
  • IntelliJ
  • MacHG
  • Murky
  • Octopus
  • Perforce
  • ScriptDebugger
  • Slender (?)
  • Smart Git
  • SourceTree
  • Sprout
  • Sublime Text
  • SynchroSVN
  • TextMate
  • TextWrangler
  • Titanium Studio
  • Tower
  • Transmit
  • vim
  • Yummy FTP
  • zsh

Scenario Testing

This section focuses on user stories and what they may try to do with our tools. Give them a go, push them hard and see if you can find our weak points. Get dirty and add and expand with your own uses and use cases.

Arment

Wants to use Kaleidoscope's folder diffing to clean up a backup so they can see what was on her older machine that they might want on her new machine, including apps and music and their library folder.

Bil

Wants FTP into a web server and use Kaleidoscope to update the website for all the stuff that is different, they don't use VCS.

Cathy

Wants to compare a project they used to track with Subversion but now they've got in a Mercurial repo. They're going to try to use VCS to do so. They might get confused along the way.

Dirk

Is copy editing and doing some large merges on a stable of very large books with lots of tiny edits. They will attempt to try and merge line-by-line.

Ellie

Will use Kaleidoscope to diff some .xibs and Xcode projects and other Xcode related files.

Fraser

Is going to use Automator and some shell script trickery to recursively diff all folders in their home directory against an external backup.

Gruber

Is going to AFP to their friend's machine over the neighbor's open WiFi network, compare their local download folder to their friend's, so they can get all the Game of Thrones episodes they might be missing before going back to class.

Hugo

Is going to use Kaleidoscope as a rsync replacement and backup tool. They have a lot of data and are tech savvy.

Irene

Is a designer with a complex hierarchy of files they maintain for theirself and clients. Instead of version control, they rely on file names and timestamps to keep track of the status of the files.

Jenneke

Finally made the move from Windows to Mac, started with a fresh user account on the mac and had someone else move all the files over but they forgot to move the MP3 collection over and they've already got stuff in iTunes and naturally, don't want to double up, they are going to try to use Kaleidoscope to compare and move files over, they are not very Mac-friendly yet.

Koen

Is going to try to use Kaleidoscope to merge a mail account by digging into the folder hierarchies and moving stuff over, and they have no idea what they are doing.

Laurens

Saves files locally with one name but moves them to a web server under a different name, the only thing that is changed is the name and file permissions, they want to use Kaleidoscope to compare and fix missing files on either side, too bad they have different names.

Mathieu

Is trying to clean up a Dropbox folder hierarchy but messed up and things are syncing they forgot about. Everything keeps changing and they're not sure what the heck is going on. Help 'em out.

Neven

While they were trying to clean up their folder structures, they somehow tried to delete the user temp files. You know, the ones Kaleidoscope needs to do it's job. Uh-oh.

Oskar

Is using Kaleidoscope to diff language strings files for a really cool new Mac OS X application. See KS through their eyes.

Pieter

Uses some weird terminal no one has ever heard of and Bazaar for SCM.

Quana

As an author Quana wants to use ks to help farm out proofing duties so that she can update her changes against her interns.

Rizzard

As a print designer Rizzard wants to check last week's ad against this week's ad so that they can see what's new before going to press.

Sherrah

As a lawyer Sherrah want to quickly scan any changes to the decree forms so that they can be summed up for her clients

Terno

As an iOS developer Terno wants to resolve team conflicts in the Mercurial repo so that they can keep the development machine humming.

Uulai

As a project planner Uulai wants to compare and updates her client's project so that she can re-use pertinent things and isn't duplicating her work.

Viglo

I can't think of a thing that they might be trying to do. I think I'm out of ideas.

Wandy
Xavieria
Yanciful
Zennographer

I may have ran out of names as well.

Internal Stakeholders Testing

Pick a team member or area and ask an interested team member what they would like tested or if they have any particular concerns about their areas (these are general areas of domain if you are looking to investigate one feature over another)

  • Doug Russell Folder scope
  • Jack Brewster Customer Support
  • Michael Gorbach UI updates
  • Michael Jurewitz Project Overseer
  • Nat Irons Build management
  • Olivier Charavel App Design
  • Phil LeTourneau UI store, help
  • Rick Fillion Sandboxing & Store
  • Sheree Peña Tester
  • Simone Manganelli Merging & VCS
  • Rob Rix Commandline shenanigans
  • Daniel Pasco Product Owner

Competitor Testing

Kaleidoscope's new merging capabilities have few options for GUI on the Mac OS X platform. Let's see how the other guys deal with both Diffing and Merging. Review their support sites and release notes for testing ideas.

Merge Tools
Diff Tools

Stress Testing

Kaleidoscope is meant to be "blazing fast" and a developer's tool. You can bet our users will put quite a load on her shoulders.

Large image files
  • Greater than 2mb - image scope is more of a fancy viewer so put her to work with the difference engine to test the masking, otherwise focus on load times in scopes
Large text files
  • Greater than 5MB up to 50GB - there are no firm limits in KS (so I am told but I have seen code and comments otherwise) and KS can take some time to get through a big comparison. try loading one, try loading two, try swapping big ones swiftly etc
  • 20K-200K line documents should be enough
Large folder hierarchies
  • Greater than 50 deep, we don't do a tree-style or crazy recursive deep penetration but just pound away at large structures and see where weaknesses can be found
Many image files
  • verify that KS can handle greater than 150, slam 'em in, swap 'em really fast, go nuts
Many text files
  • verify that KS can handle greater than 150, swap
Complex image diffs
  • hard one to peg, image diffing only gets complex during masking
  • verify that KS can handle big crazy images, get huge ones with lots of pixel-level differences and use both types of masking
Complex text diffs
  • pull from support/FB queue for difficult examples
  • single word lines and lots of 'em (good example from rick on FB)
Complex folder diffs
  • heavy files, deep hierarchy, lots of content,
Complex saves
  • see merging?
Large Changesets
  • verify that KS can handle greater than 1500 changes in set with ease
Rapid usage
  • Get fast. Get furious. Get rowdy. Hire a monkey that deals in chaos.

Feature & Function Testing

The next section attempts to navigate the user interface, the command line tools, the features and the application in general. We are attempting to discern whether each works and looks as expected.

General Interface

First Launch

Licensing will be something encountered on launches but is to be covered under the Licensing & Registration testing subsection.

  • verify license agreement is not shown to user on first launch recently removed
    • UNLESS - if the terms changed, Kaleidoscope should show you a diff and then allow you to accept/reject -- this is something we botched in the past due to changed bundle identifiers -- don't let this slip by again!
  • verify Getting Started window opens
    • see Getting Started Window Section below for specific things to poke at for that feature [seeing a massive overhaul currently]
  • verify you are not pummeled with too many windows and chaos upon launch
    • ex - ksdiff/integration nag, registration nag, empty drop area, other Kaleidoscope notices - there are many

Getting Started Window

This window will be seen on initial launch and each launch after provided the option is toggled for the user. It functions as a welcome message, tour, sampler, informational, and email sign up list.

  • panel
    • verify window placement
    • verify general interactivity
    • verify can hide and reshow
  • navigation
    • verify disc navigation
    • verify mouse and swipe navigation & animations
  • examples
    • verify example is shown, is clickable, has hover, matches itself, matches actual opened diff [see examples subsection for testing]
  • retina/non
    • verify everything in view is retina friendly
  • interaction
    • verify each and every clickable item. this screen has gotten tripped up on 10.7 retina before, test and poke all of it
  • animations
    • verify navigation and examples animations happen and are smooth
  • open launch behavior
    • verify toggling respects value upon relaunch
  • email sign up
    • we may kill the auto-fill and it's undetermined whether or not we have decided to have the opt-in toggle defaulted in the on or off position
Example Files from Getting Started window

KS getting started shows you example files. These need to be able to be used just like any other files but also re-created fresh for each use.

  • verify the sample files open
  • verify the sample files are savable
  • verify the files are fresh on retry
  • verify that modifying one set does not modify the other
  • verify the files are removed locally on relaunch
About Window
  • verify that both "acknowledgements" and "license agreement" buttons work
  • verify that build number, version number, copyright is correct
    • These should be correct for internal builds
  • ensure about window displays MAS/direct details as apropos

Help Guide

  • verify Help Doc is bundled
  • verify search ability
  • verify current version information
  • verify image assets on retina and non
  • verify old images are gone and new ones in place
  • verify copy as much as you are able, get an editor
  • verify all external links
  • verify copy has been proofread
  • verify and fact check all content, walk through the entire guide
  • verify this will match what is publicly available online

FastSpring & Registration

  • make sure to test in following conditions:
    • guest user account
    • admin user account
    • non-admin user account
  • verify that demo expires
  • verify demo or beta notice shows in all views and scopes (beyond drop window)
  • register in demo period
  • register after demo expiration
  • can beta builds be purchased from successfully
  • use alternate licenses - if those will exist!?
  • attempt to trick Kaleidoscope into restarting demo
  • verify temp license date expiration is displayed accurately
  • verify KS1 settings do not interfere with KS2 settings
  • verify MAS license can be used with Direct Sale copy
  1. A virgin MAS build will exit with code -73 on launch, causing the app to request MAS credentials.
  2. Once MAS credentials are given, the app should run. There should be a _MASReceipt dir in the bundle.
  3. Once running, the app should have copied the MASReceipt into App Support
  4. Deleting the MASReceipt from app support and rerunning the app should recopy it there
  5. Running the DS version after having run the MAS version and gotten a MASReceipt, you should be able to run as if you were licensed, and the reg screen should indicate this.

Sandboxing & Security Support

All our builds, public & private should be Gatekeeper signed from here to eternity. Sandboxing of KS2 is not enabled for our direct-sale offerings. The MAS published version of KS2 will be Sandboxed as well as offer the ksdiff commandline tool as a separate download.

  • Developer signed
    • verify each build is signed and behaves well with Gatekeeper
    • Note: GK is on 10.7.5 and up as of Sept 2012
  • Sandboxing
    • monitor Console.app for sandboxd and related messages while performing your regular usage and testing tasks

Updating, Installation & Uninstalling

KS1 had used it's own Sparkle-style updater but KS2 DS has opted for Sparkle usage. Review setup here

Direct sale release
  • verify you are alerted to an update and the alert looks and behaves the way you expect
  • verify that the release notes are as expected
  • verify that you can install the update over the app
  • verify that you can launch and use the update
  • verify notice of changed terms if applicable
  • verify data in About page and Get Info in the Finder
Sparkle End User Updating
  • verify updates work when app is stored outside of /Applications
  • verify updates work via helper apps like AppFresh/Bodega
  • verify updates work on early 10.8 -- see CFNetwork thing that affected NetNewsWire/MarsEdit
  • verify updates work after a force down
  • verify updates work with gatekeeper as expected
  • verify updates work when installed and updated from: guest/admin/nonadmin
  • verify updates work over iffy network
  • verify updates work on old 10.7s
  • verify updates work on old 10.8s
  • verify updates work with proxy settings non-standard
  • verify updates work when rename the app ala "Kaleidoscope-BETA"
  • verify updates work when run alongside KS1 and don't overwrite KS1 erroneously
  • verify updates work when run over filevault
  • verify updates work when theUnarchiver gets in the middle (known signing mangler)
  • verify updates work when update from another account (insalled via admin, can update via guest)
  • verify updates work when Application Support docs/folder has non-standard permissions
Direct's enclosed ksdiff
  • verify KS detects an old ksdiff and prompts for an update but not too much
  • verify the update is installed and the old ksdiff removed including helpers
MAS release
  • verify MAS purchases can update from MAS I don't know how to ensure this
MAS ksdiff external/manual install
  • verify the download runs and includes the installer and uninstaller
  • verify the license is current
  • verify the images/icons are correct on retina and non
  • verify gatekeeper status if that exists

Crash Reporting

  • make sure you have a non-harmful way to crash
    • pluck reproducible crashers from FogBugz
  • verify that you can successfully send a crash report
    • verify that you received the crash report at Hockey App
  • verify pertinent info is incoming along with crash report
  • verify with HA admin that dSYMs are symbolicating
  • verify FB/HA integration performing as expected

Preferences

General
  • verify that every option does what it says it will do
  • verify assets
  • verify spelling
  • verify interactivty of all settings
Image
  • verify navigator choices respected
  • verify view type is reused from last viewed image to on to the next image diff
  • verify background color enabled and chooser color changes along with selection
  • verify transparency settings, all options
  • very important: make sure the absolute/proportional image scope difference rendition works properly
  • verify same/diff coloring settings
Folder
  • verify hidden files settings respected (monitor for kscache)
  • verify copy files dialog settings honored
Text
  • verify light/dark performs as expected & elements aren't busted
  • verify coloring toggles & reflects & maintains
  • verify Font choices respected & manipulations shown live
  • verify wrapping performs as expected

Integration

Install & Configure
  • install and uninstall ksdiff
    • which ksdiff verify = /usr/local/bin/ksdiff
  • Configuring of command-line tool
    • Installation of Versions - no toggle shown
    • Installation of Subversion - no toggle shown
    • Installation of Perforce - no toggle shown
    • Installation of Git - should toggle
    • Installation of Mercurial - no toggle shown
    • Installation of Bazaar - should toggle
    • Installation of TextMate svn bundle - no toggle shown
    • Installation of TextMate Mercurial bundle - no toggle shown
  • verify behavior when old Kaleidoscope defaults are in place for a particular VCS
  • new "feature" uninstalling ksdiff also unsets VCS integration
Integration Window
  • verify integration window shows correct installation status
  • verify typos and grammar
  • verify examples syntax
  • verify links are correct
  • verify window behavior

Tabs

  • verify that tab name is displayed (normal comparison, changeset from VCS, changeset from GUI)
  • create and close
  • verify key equivalents
  • re-order tabs
  • shelf toggle
    • verify that counter works
    • verify that button show/hides shelf [they might remove this!]
  • verify overflow menu, create many tabs
  • verify new tab button
  • verify behavior on moving in/out of Full Screen

Path Bar

  • verify that path is shown
  • verify expanding on hover
  • verify "keep open"
  • verify collapsing and truncation on window resize
  • verify swap button
  • verify key equivalents
  • verify all pulldown menus
  • verify path bar tooltips
  • verify coloring
  • verify animations
  • verify behavior on window resize
  • verify behavior on moving in/out of Full Screen
  • verify new 1-item loaded rules are followed for a/b (ie, no swap/placeholder)

File Shelf AKA File Drawer

  • verify mouse selection

  • verify remove

  • verify reorder

  • verify reveal

  • verify CMI right click offers to add

  • verify CMI on item allows removal

  • verify animation on removal

  • verify double click empty area invoke add file dialog

  • verify scrolling, drop in many files

  • verify step through, both mouse and keyboard

  • verify swapping, swapping in mass, swiftly, etc

  • verify retina assets

  • verify item drag bounds scrolling

  • verify scrollbars

  • verify animations

  • verify behavior in various states

  • verify variety of supported file formats (text, images)

  • verify new 1-item loaded rules are followed for a/b

  • verify view behavior, of every view

  • verify behavior on moving in/out of Full Screen

  • drag and drop

    • verify single file
    • verify multiple files (same file type)
    • verify multiple files (supported and unsupported)
    • verify ignore sheet

Shortcuts & Menu Items Testing

Has recently been in massive flux. The original audit is null and void. See Dropbox & Kaleidoscope for most current status.

Animations Testing

First a census, then track when they should appear. Then we attempt to trigger them and monitor their performance.

  • Known animations
    • drop window expand
    • drop window contract
    • hover glow on drop window
    • file shelf
    • getting started example popouts [may be removed]
    • 3-up open
  • test
    • performance
    • retina v. non
    • 10.7 v. 10.8
    • perform on bare bones supported OS

Gestures Testing

First a census, then track where they are available. Then see if they are all behaving well. Make sure we cover the settings differences in systems preferences.

  • audit
    • diff selection in text view? (may have been killed, unclear, bug on file)
    • image navigation pinch & zoom
  • test
    • performance
    • perform on bare bones supported OS

Comparisons

Invoking

Kaleidoscope should behave as follows with respect to new windows and tabs:

  1. Command-N opens new window. Command-T opens new tab. Both are in the File menu.
  2. Clicking on an example in the Getting Started window will close any empty launch windows and open the example in a new window, allowing us to do our nice animation "pop out" of the example document.
  3. Opening anything from the command line (ksdiff or VCS) will always open a new window, closing any launch windows.
  4. Files dragged onto the dock icon will always open in a new window (closing any launch windows).
  5. Files dragged onto a launch window will open in that launch window, using the "expand" animation.
  6. Files opened using File -> Open or using the open panel obtained by clicking in the launch window will also open in that launch window, using the "expand" animation.
  7. Comparisons "spawned" from other comparison documents will open in a tab next to the parent document. Currently, this only applies to drilling down in folder diff.
  8. Diffs opened using the Open Recents menu item should open in any existing launch windows using the "expand" animation
  9. Dragging arbitrary text or images should invoke a new comparison if the filetype does not match the current, or added to the current set if they do

When diffing app bundles:

  • if the user checks off "Do not ask me again" it doesn't appear next time.
  • if the user selects "Cancel" the merge choice should get cleared out
  • if the user selects "Keep Selection", the merge choice should remain. If the user undoes that merge choice (via cmd-z, or manually cycling through the options back to none), then makes the same, or another merge choice, the dialog should no longer appear (so it can come up a maximum of once per document instantiation).
  • check to make sure it works with multi-select.
Invoking Changeset Comparisons

Basics

  • verify all VCSs supported
    • Git, Subversion, Mercurial, Bazaar, Perforce
  • make sure you have ksdiff installed, verify in Integration set up
  • verify keyboard focus is remembered while switching tabs
  • open a very large changeset
    • expected, long list of files to appear item by item
  • verify that tab title is properly displayed
  • verify that origin icon in header is properly displayed
  • verify that paths are properly displayed
  • verify that tmp files aren't editable
Changesets sent from apps

Marketing lists some apps by name and we hope that more and more will integrate. There are a host of apps that have us on file as a difftool but won't quite yet for merging.

  • Do apps behave the same as CLI, are there any oddities
  • Do apps show correct window title & icon and can they for sandboxed apps [no?]?
    • Tower (important!)
    • Versions (crucial!)
    • SourceTree (important!)
    • Gitbox (important!)
    • MacHG (was broken for KS1)
Changesets sent from command-line

[ALL OF THIS HAS CHANGED RECENTLY] You'll want to try invoking every which way there is. Below is a small list of commons ways to diff a thing with that particular VCS. Send changesets and diffs in each and every configuration you can think of. For example, you can send bzr commands to svn and git repos. Try patch and diff files. Do it over the network or not. Get weird.

Subversion

http://svnbook.red-bean.com/en/1.0/re09.html

  • wc head to base
  • revision to head
  • revision to revision
  • range of revisions
  • rev to rev specify trunk
  • rev to rev by file name
  • with --diff-cmd flags
  • wc head to branch head
  • rev to branch head
  • head to tag
  • rev changeset
Git

http://kernel.org/pub/software/scm/git/docs/git-diff.html See also git clone https://github.com/rustle/KaleidoscopeVCSTidbits.git

  • head to base
  • by SHA
  • SHA to SHA
  • range of SHA
  • SHA to branch
  • filename?
  • what flags?
  • head to branch
  • head to tag
  • tag to tag
  • by SHA filename
  • commit to head
  • head to branch
  • commit to branch
  • commit to commit
  • commit to commit by range
Mercurial

http://mercurial.selenic.com/wiki/MergeProgram http://mercurial.selenic.com/wiki/ExtdiffExtension

  • by commit,
  • head/base,
  • remote
  • filename
Bazaar

http://doc.bazaar.canonical.com/latest/en/user-reference/diff-help.html

  • head to base
  • rev to rev
  • rev to rev range
  • head to rev
  • head by file
  • head to file by branch
  • head to branch
  • file to file by branch
# Perforce

http://www.perforce.com/perforce/r12.1/manuals/cmdref/diff.html

  • by file
  • by date
  • by date range
  • by patch?
  • by revision
  • by revision range
TextMate 1 svn
  • see svn
TextMate 1 mercurial
  • see hg
TextMate 2
  • plugins not found
Testing Special File Types

There are sniffers but let's take a look at special instances.

  • RTF - should proxy to plaintext
  • Word - should be read as RTF which proxies to plain text
  • PDF - not supported but we may begin showing as image? [this is slated for later]
  • AI - should show not-supported
  • SVG - should show as text diff [although there has been talk about offering to view as image]

Image Scope

Here we are focusing on the comparisons only. Please see merging subsection for that feature. (VCS image "merging" to be implemented in a future update)

  • verify zooming and moving

  • verify navigator behavior and settings

  • verify differently sized files

  • verify retina behavior

  • verify dragging

  • verify pinch zoom

  • verify preferences

  • scope toolbar

    • verify all view options, and sub options
    • verify items work and behave as expected
    • verify item overflow behavior
    • verify items are correctly enabled/disabled
  • drag and drop
    • verify single file
    • verify multiple files (same file type)
    • verify multiple files (supported and unsupported)
    • verify ignore sheet

Text Scope

Here we are focusing on the comparisons only. Please see merging subsection for that feature.

  • verify emoji and non-standard characters
  • verify diff result
  • verify change stepper, mouse, trackpad and keyboard
  • verify search, results
  • verify search, formatting and drawing
  • verify scrollbar behavior
  • verify scrollbar change location indicators
  • verify smooth scrolling issues
  • verify coloring
  • verify lives changes of prefs
  • verify selection
  • scope toolbar
    • verify all view options, and sub options
    • verify items work and behave as expected
    • verify item overflow behavior
    • verify items are correctly enabled/disabled
  • drag and drop
    • verify single file
    • verify multiple files (same file type)
    • verify multiple files (supported and unsupported)
    • verify ignore sheet
Whitespace & Line Endings
  • verify CRLF
  • verify CR
  • verify LF
  • verify whitespace

Folder Scope

This TLF is new and needs your special care and attention. Break it.

  • verify dragging, dropping, opening, adding, ksdiffing ETC all ways of initiation a new diff

  • verify sorting

  • verify each view

  • verify change highlighting

  • verify toolbar tools

  • verify content changes,

  • verify file renaming, repeatedly

  • verify removing items, repeatedly

  • verify adding items, repeatedly

  • verify modification of creation/modified values

  • verify related shortcuts

  • verify related menu items

  • verify right-click menu items every where

  • verify viewing vs merging status

  • scope toolbar

    • verify all view options, and sub options
    • verify items work and behave as expected
    • verify item overflow behavior
    • verify items are correctly enabled/disabled
  • drag and drop
    • verify single file
    • verify multiple files (same file type)
    • verify multiple files (supported and unsupported)
    • verify ignore sheet

Merging & Undo by Scope

Text Merging

  • From the example text diff in the Getting Started window with each type of change (insert, delete, modify), perform each of (merge left, merge right, both merge left first, both merge right first). After each merge, make sure the document is at expected state. (Both merges shouldn't work on inserts and deletes, so you'll only have 8 checks to do here.)
  • Make sure both merges are disabled for inserts and deletes. The both merge buttons (hold down the option key to display them) and the merge both menu items in the Merge menu should be disabled.
  • Save. For 2-up view, make sure both files A & B on disk are synchronized with the state displayed in the app. If either A or B has not changed from its last saved version, it should not have been touched on disk.
  • Repeat the above tests for 3-up view. (Need good example conflicted test repo or SHAs on Kaleidoscope branches to merge that produce conflicts for testing.) Saving in 3-up view should only save the merged file to the location specified by the VCS tool. Neither file A or file B should be touched when saving.
Text Merging Undo
  • From the example text diff in the Getting Started window with each type of change (insert, delete, modify), perform each of (merge left, merge right, both merge left first, both merge right first). After each merge, undo the merge by pressing Cmd-Z, and make sure the document is back in its original form. (Both merges shouldn't work on inserts and deletes, so you'll only have 8 checks to do here.)
  • Perform a series of multiple merges on different changes: (e.g.: merge left on change 1, merge right on change 2, both merge left first on change 3), and then undo them all. Make sure document is at expected state at each state of undo.
  • Switch files in the file shelf. After saving or discarding changes, you should not be able to undo once the switch is complete.

Text Editing & Text VCS Conflict Resolution

arbitrary text editing is a sub feature of merging which you will only see during a 3-up merge which is currently only going to be invoked via VCSs git and hg. the same do/undo stack rules apply here

  • verify KS can accept the mergetool request
  • verify multiple ways of sending the conflict
  • verify arbitrary merges
  • verify KS shows appropriate UI
  • verify undo works at all
  • verify key commands and menu items
  • verify arbitrary text and pulling text from one side or the other works with undo
  • verify changing view types doesn't alter the stack or results thereof
  • verify text editing works with 2-step characters & undo stack
  • verify emoji and other weird characters works in text editing and the undo stack

Folder Merging & Undo Support

  • verify arrows during swapping
  • verify copy behaves as expected
  • verify preferences are honored
  • verify undo support works
  • verify key commands and menu items
  • verify a/b swap behavior
  • verify filter behavior

Image VCS Conflict Resolution

Git and Mercurial will allow you to choose a file when a merge conflict happens on an image. When sending to Kaleidoscope we offer a nice selection UI to resolve this scenario.

  • verify KS can accept the merge type
  • verify KS shows appropriate ui for image
  • verify KS mine/theirs ui makes sense and flows like text
  • verify KS interactivity when committing the change
  • verify KS can handle appropriate view type changes during compare
  • verify KS can handle disk changes in the scenario
  • verify choosing and commiting is properly reflected in the repo
  • verify backing out leaves repo in expected state

Other things

This is an unorganized list of things not fleshed out quite yet.

  • Mergetool Integration with Existing VCS guis
    • someone is going to have to reach out to Cornerstone et al
  • Easter Eggs
    • we should have them and they should be great
    • if you try to diff Kaleidoscope with Kaleidoscope you should get a thing
    • if you remove 2 files referenced in a 3 up, I think we could reuse the middle pane to put a secret thing.
  • typos & grammatical errors
    • is there a way to do any of that programmatically?
  • obvious design errors
    • vs your own eyeballs but also against comps?
  • assets misplaced, unused, etc
  • animations audit
    • is there anyway to search the project for strings to get a good idea of the ones that there are?
  • missing retina assets
    • will the help guide pick these up correctly?
  • issues when moving from retina to non screen
  • weird things showing up in the Console
    • especially sandboxen
  • behavior and results using the same test data with similar apps
  • Merging RTF and Word docs
    • you can sort of do it but it's… weird.
  • Gestures
    • I need to know where we look for these so we can audit the full set of supported gestures
  • accessibility
    • what has been baked in and addressed?
    • beyond voiceover?
    • colorblindness testing [Xscope is awesome for this]
    • is everything perceivable [over arching view]
    • is everything operable
    • is everything understandable [more detailed view]
    • is it a robust experience
    • latest and greatest updates, do we use them
  • Product vs Marketing
    • compare & contrast Kaleidoscope marketing site against Kaleidoscope
    • help guide and about page and getting started should all also look like the site
  • Bare bones regression
    • simply compare diffs in ks1 against diffs in ks2
  • VCS collision?
    • VCS integration meant for KS interfering with VCS or other tools integration
  • Navigation Key Commands
    • by context [scope/build type/items loaded/merge state/VCS state]? these are in flux.
    • needs updating
  • FileTypes
    • there is a list, somewhere? how about a NOT supported list? ai, pdf [doug may well change that]
  • Encoding
    • I've been using emoji and weird characters but filetype is fucking frightening.
  • Binary Auditing
    • spelunking in the binaries for things we don't want there
  • Artifacts and Detritus
    • spelunking into the syspref and app support detritus for anything hinky
  • Slender.app audit?
    • Slender just crashes constantly for me
  • Enabling various keyboards

❚❚❚ So long and thanks for all the fish.